2023-02-12 14:59:59,029 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3 2023-02-12 14:59:59,042 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-12 14:59:59,056 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-12 14:59:59,057 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00, deleteOnExit=true 2023-02-12 14:59:59,057 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-12 14:59:59,057 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/test.cache.data in system properties and HBase conf 2023-02-12 14:59:59,058 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/hadoop.tmp.dir in system properties and HBase conf 2023-02-12 14:59:59,058 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/hadoop.log.dir in system properties and HBase conf 2023-02-12 14:59:59,058 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-12 14:59:59,059 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-12 14:59:59,059 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-12 14:59:59,177 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-12 14:59:59,567 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-12 14:59:59,572 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-12 14:59:59,573 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-12 14:59:59,573 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-12 14:59:59,573 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-12 14:59:59,574 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-12 14:59:59,574 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-12 14:59:59,574 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-12 14:59:59,575 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-12 14:59:59,575 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-12 14:59:59,575 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/nfs.dump.dir in system properties and HBase conf 2023-02-12 14:59:59,576 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/java.io.tmpdir in system properties and HBase conf 2023-02-12 14:59:59,576 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-12 14:59:59,576 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-12 14:59:59,577 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-12 15:00:00,052 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-12 15:00:00,056 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-12 15:00:00,326 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-12 15:00:00,478 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-12 15:00:00,492 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-12 15:00:00,529 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-12 15:00:00,559 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/java.io.tmpdir/Jetty_localhost_42999_hdfs____.jp9tob/webapp 2023-02-12 15:00:00,682 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42999 2023-02-12 15:00:00,718 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-12 15:00:00,718 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-12 15:00:01,097 WARN [Listener at localhost/35339] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-12 15:00:01,197 WARN [Listener at localhost/35339] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-12 15:00:01,214 WARN [Listener at localhost/35339] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-12 15:00:01,220 INFO [Listener at localhost/35339] log.Slf4jLog(67): jetty-6.1.26 2023-02-12 15:00:01,224 INFO [Listener at localhost/35339] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/java.io.tmpdir/Jetty_localhost_40839_datanode____1s3ybt/webapp 2023-02-12 15:00:01,320 INFO [Listener at localhost/35339] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40839 2023-02-12 15:00:01,608 WARN [Listener at localhost/45403] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-12 15:00:01,622 WARN [Listener at localhost/45403] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-12 15:00:01,627 WARN [Listener at localhost/45403] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-12 15:00:01,630 INFO [Listener at localhost/45403] log.Slf4jLog(67): jetty-6.1.26 2023-02-12 15:00:01,636 INFO [Listener at localhost/45403] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/java.io.tmpdir/Jetty_localhost_37743_datanode____.cunton/webapp 2023-02-12 15:00:01,740 INFO [Listener at localhost/45403] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37743 2023-02-12 15:00:01,758 WARN [Listener at localhost/44453] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-12 15:00:01,779 WARN [Listener at localhost/44453] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-12 15:00:01,782 WARN [Listener at localhost/44453] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-12 15:00:01,784 INFO [Listener at localhost/44453] log.Slf4jLog(67): jetty-6.1.26 2023-02-12 15:00:01,791 INFO [Listener at localhost/44453] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/java.io.tmpdir/Jetty_localhost_45637_datanode____.9kbfkw/webapp 2023-02-12 15:00:01,906 INFO [Listener at localhost/44453] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45637 2023-02-12 15:00:01,914 WARN [Listener at localhost/38323] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-12 15:00:02,093 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xdbde5c62178ce378: Processing first storage report for DS-cc807189-4c44-4066-9b81-0a7d2acae137 from datanode 7dc5048d-e841-4f47-b47b-e95ce3c57560 2023-02-12 15:00:02,094 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xdbde5c62178ce378: from storage DS-cc807189-4c44-4066-9b81-0a7d2acae137 node DatanodeRegistration(127.0.0.1:33343, datanodeUuid=7dc5048d-e841-4f47-b47b-e95ce3c57560, infoPort=44139, infoSecurePort=0, ipcPort=45403, storageInfo=lv=-57;cid=testClusterID;nsid=1513109252;c=1676214000132), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-12 15:00:02,095 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf0698cab6b7294bf: Processing first storage report for DS-e50680d0-d606-4c5b-bdce-8317dc34931c from datanode ea05670d-b581-4fa9-ad56-726719504e8a 2023-02-12 15:00:02,095 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf0698cab6b7294bf: from storage DS-e50680d0-d606-4c5b-bdce-8317dc34931c node DatanodeRegistration(127.0.0.1:39973, datanodeUuid=ea05670d-b581-4fa9-ad56-726719504e8a, infoPort=42213, infoSecurePort=0, ipcPort=38323, storageInfo=lv=-57;cid=testClusterID;nsid=1513109252;c=1676214000132), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-12 15:00:02,095 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x95feed7935824b89: Processing first storage report for DS-308f6b74-90b2-4af6-b9b4-9f0dbec0aee8 from datanode b02a920e-a537-4a3b-8680-c32dd40f546d 2023-02-12 15:00:02,095 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x95feed7935824b89: from storage DS-308f6b74-90b2-4af6-b9b4-9f0dbec0aee8 node DatanodeRegistration(127.0.0.1:46017, datanodeUuid=b02a920e-a537-4a3b-8680-c32dd40f546d, infoPort=43865, infoSecurePort=0, ipcPort=44453, storageInfo=lv=-57;cid=testClusterID;nsid=1513109252;c=1676214000132), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-12 15:00:02,095 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xdbde5c62178ce378: Processing first storage report for DS-6e9cf47f-19f2-430c-95f7-b148c62d875d from datanode 7dc5048d-e841-4f47-b47b-e95ce3c57560 2023-02-12 15:00:02,095 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xdbde5c62178ce378: from storage DS-6e9cf47f-19f2-430c-95f7-b148c62d875d node DatanodeRegistration(127.0.0.1:33343, datanodeUuid=7dc5048d-e841-4f47-b47b-e95ce3c57560, infoPort=44139, infoSecurePort=0, ipcPort=45403, storageInfo=lv=-57;cid=testClusterID;nsid=1513109252;c=1676214000132), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-12 15:00:02,095 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf0698cab6b7294bf: Processing first storage report for DS-c7fd88b1-17e2-4d84-95be-1850d0373ac0 from datanode ea05670d-b581-4fa9-ad56-726719504e8a 2023-02-12 15:00:02,095 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf0698cab6b7294bf: from storage DS-c7fd88b1-17e2-4d84-95be-1850d0373ac0 node DatanodeRegistration(127.0.0.1:39973, datanodeUuid=ea05670d-b581-4fa9-ad56-726719504e8a, infoPort=42213, infoSecurePort=0, ipcPort=38323, storageInfo=lv=-57;cid=testClusterID;nsid=1513109252;c=1676214000132), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-12 15:00:02,096 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x95feed7935824b89: Processing first storage report for DS-06667344-7717-4084-bc5c-d8b09798a1b3 from datanode b02a920e-a537-4a3b-8680-c32dd40f546d 2023-02-12 15:00:02,096 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x95feed7935824b89: from storage DS-06667344-7717-4084-bc5c-d8b09798a1b3 node DatanodeRegistration(127.0.0.1:46017, datanodeUuid=b02a920e-a537-4a3b-8680-c32dd40f546d, infoPort=43865, infoSecurePort=0, ipcPort=44453, storageInfo=lv=-57;cid=testClusterID;nsid=1513109252;c=1676214000132), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-12 15:00:02,301 DEBUG [Listener at localhost/38323] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3 2023-02-12 15:00:02,361 INFO [Listener at localhost/38323] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/zookeeper_0, clientPort=61893, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-12 15:00:02,376 INFO [Listener at localhost/38323] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=61893 2023-02-12 15:00:02,386 INFO [Listener at localhost/38323] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 15:00:02,388 INFO [Listener at localhost/38323] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 15:00:03,057 INFO [Listener at localhost/38323] util.FSUtils(479): Created version file at hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b with version=8 2023-02-12 15:00:03,057 INFO [Listener at localhost/38323] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/hbase-staging 2023-02-12 15:00:03,355 INFO [Listener at localhost/38323] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-12 15:00:03,838 INFO [Listener at localhost/38323] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=18 2023-02-12 15:00:03,870 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 15:00:03,870 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-12 15:00:03,871 INFO [Listener at localhost/38323] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-12 15:00:03,871 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 15:00:03,871 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-12 15:00:04,013 INFO [Listener at localhost/38323] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-12 15:00:04,094 DEBUG [Listener at localhost/38323] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-12 15:00:04,188 INFO [Listener at localhost/38323] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45369 2023-02-12 15:00:04,198 INFO [Listener at localhost/38323] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 15:00:04,200 INFO [Listener at localhost/38323] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 15:00:04,219 INFO [Listener at localhost/38323] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45369 connecting to ZooKeeper ensemble=127.0.0.1:61893 2023-02-12 15:00:04,262 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:453690x0, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-12 15:00:04,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:45369-0x100490bcc700000 connected 2023-02-12 15:00:04,289 DEBUG [Listener at localhost/38323] zookeeper.ZKUtil(164): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-12 15:00:04,290 DEBUG [Listener at localhost/38323] zookeeper.ZKUtil(164): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 15:00:04,294 DEBUG [Listener at localhost/38323] zookeeper.ZKUtil(164): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-12 15:00:04,301 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45369 2023-02-12 15:00:04,302 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45369 2023-02-12 15:00:04,302 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45369 2023-02-12 15:00:04,302 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45369 2023-02-12 15:00:04,303 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45369 2023-02-12 15:00:04,308 INFO [Listener at localhost/38323] master.HMaster(439): hbase.rootdir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b, hbase.cluster.distributed=false 2023-02-12 15:00:04,378 INFO [Listener at localhost/38323] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-12 15:00:04,378 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 15:00:04,378 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-12 15:00:04,378 INFO [Listener at localhost/38323] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-12 15:00:04,379 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 15:00:04,379 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-12 15:00:04,383 INFO [Listener at localhost/38323] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-12 15:00:04,386 INFO [Listener at localhost/38323] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37113 2023-02-12 15:00:04,388 INFO [Listener at localhost/38323] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-12 15:00:04,394 INFO [Listener at localhost/38323] mob.MobFileCache(124): MobFileCache disabled 2023-02-12 15:00:04,395 INFO [Listener at localhost/38323] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 15:00:04,397 INFO [Listener at localhost/38323] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 15:00:04,398 INFO [Listener at localhost/38323] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37113 connecting to ZooKeeper ensemble=127.0.0.1:61893 2023-02-12 15:00:04,406 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:371130x0, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-12 15:00:04,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:37113-0x100490bcc700001 connected 2023-02-12 15:00:04,407 DEBUG [Listener at localhost/38323] zookeeper.ZKUtil(164): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-12 15:00:04,408 DEBUG [Listener at localhost/38323] zookeeper.ZKUtil(164): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 15:00:04,409 DEBUG [Listener at localhost/38323] zookeeper.ZKUtil(164): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-12 15:00:04,410 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37113 2023-02-12 15:00:04,410 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37113 2023-02-12 15:00:04,410 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37113 2023-02-12 15:00:04,411 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37113 2023-02-12 15:00:04,411 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37113 2023-02-12 15:00:04,423 INFO [Listener at localhost/38323] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-12 15:00:04,423 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 15:00:04,423 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-12 15:00:04,424 INFO [Listener at localhost/38323] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-12 15:00:04,424 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 15:00:04,424 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-12 15:00:04,424 INFO [Listener at localhost/38323] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-12 15:00:04,426 INFO [Listener at localhost/38323] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33845 2023-02-12 15:00:04,426 INFO [Listener at localhost/38323] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-12 15:00:04,427 INFO [Listener at localhost/38323] mob.MobFileCache(124): MobFileCache disabled 2023-02-12 15:00:04,428 INFO [Listener at localhost/38323] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 15:00:04,430 INFO [Listener at localhost/38323] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 15:00:04,432 INFO [Listener at localhost/38323] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33845 connecting to ZooKeeper ensemble=127.0.0.1:61893 2023-02-12 15:00:04,438 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:338450x0, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-12 15:00:04,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33845-0x100490bcc700002 connected 2023-02-12 15:00:04,439 DEBUG [Listener at localhost/38323] zookeeper.ZKUtil(164): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-12 15:00:04,440 DEBUG [Listener at localhost/38323] zookeeper.ZKUtil(164): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 15:00:04,441 DEBUG [Listener at localhost/38323] zookeeper.ZKUtil(164): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-12 15:00:04,442 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33845 2023-02-12 15:00:04,442 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33845 2023-02-12 15:00:04,443 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33845 2023-02-12 15:00:04,443 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33845 2023-02-12 15:00:04,443 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33845 2023-02-12 15:00:04,462 INFO [Listener at localhost/38323] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-12 15:00:04,462 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 15:00:04,463 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-12 15:00:04,463 INFO [Listener at localhost/38323] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-12 15:00:04,463 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-12 15:00:04,463 INFO [Listener at localhost/38323] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-12 15:00:04,463 INFO [Listener at localhost/38323] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-12 15:00:04,465 INFO [Listener at localhost/38323] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38453 2023-02-12 15:00:04,465 INFO [Listener at localhost/38323] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-12 15:00:04,466 INFO [Listener at localhost/38323] mob.MobFileCache(124): MobFileCache disabled 2023-02-12 15:00:04,467 INFO [Listener at localhost/38323] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 15:00:04,470 INFO [Listener at localhost/38323] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 15:00:04,472 INFO [Listener at localhost/38323] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38453 connecting to ZooKeeper ensemble=127.0.0.1:61893 2023-02-12 15:00:04,477 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:384530x0, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-12 15:00:04,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:38453-0x100490bcc700003 connected 2023-02-12 15:00:04,478 DEBUG [Listener at localhost/38323] zookeeper.ZKUtil(164): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-12 15:00:04,479 DEBUG [Listener at localhost/38323] zookeeper.ZKUtil(164): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 15:00:04,480 DEBUG [Listener at localhost/38323] zookeeper.ZKUtil(164): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-12 15:00:04,480 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38453 2023-02-12 15:00:04,481 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38453 2023-02-12 15:00:04,481 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38453 2023-02-12 15:00:04,482 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38453 2023-02-12 15:00:04,482 DEBUG [Listener at localhost/38323] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38453 2023-02-12 15:00:04,483 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,45369,1676214003194 2023-02-12 15:00:04,492 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-12 15:00:04,494 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,45369,1676214003194 2023-02-12 15:00:04,512 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-12 15:00:04,512 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-12 15:00:04,512 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 15:00:04,512 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-12 15:00:04,512 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-12 15:00:04,514 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-12 15:00:04,515 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,45369,1676214003194 from backup master directory 2023-02-12 15:00:04,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-12 15:00:04,519 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,45369,1676214003194 2023-02-12 15:00:04,520 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-12 15:00:04,520 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-12 15:00:04,520 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,45369,1676214003194 2023-02-12 15:00:04,523 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-12 15:00:04,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-12 15:00:04,613 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/hbase.id with ID: 30424d8b-ad56-47a0-8406-ccf64d20b7a4 2023-02-12 15:00:04,656 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-12 15:00:04,673 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 15:00:04,727 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x21deb758 to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:04,760 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6bcec4e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:04,783 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-12 15:00:04,784 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-12 15:00:04,801 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-12 15:00:04,801 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-12 15:00:04,803 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-12 15:00:04,807 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-12 15:00:04,808 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-12 15:00:04,841 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/MasterData/data/master/store-tmp 2023-02-12 15:00:04,928 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:04,929 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-12 15:00:04,929 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 15:00:04,929 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 15:00:04,929 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-12 15:00:04,930 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 15:00:04,930 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 15:00:04,930 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676214004929Disabling compacts and flushes for region at 1676214004929Disabling writes for close at 1676214004929Writing region close event to WAL at 1676214004930 (+1 ms)Closed at 1676214004930 2023-02-12 15:00:04,931 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/MasterData/WALs/jenkins-hbase4.apache.org,45369,1676214003194 2023-02-12 15:00:04,952 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45369%2C1676214003194, suffix=, logDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/MasterData/WALs/jenkins-hbase4.apache.org,45369,1676214003194, archiveDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/MasterData/oldWALs, maxLogs=10 2023-02-12 15:00:05,006 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46017,DS-308f6b74-90b2-4af6-b9b4-9f0dbec0aee8,DISK] 2023-02-12 15:00:05,006 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39973,DS-e50680d0-d606-4c5b-bdce-8317dc34931c,DISK] 2023-02-12 15:00:05,006 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33343,DS-cc807189-4c44-4066-9b81-0a7d2acae137,DISK] 2023-02-12 15:00:05,014 DEBUG [RS-EventLoopGroup-5-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-12 15:00:05,093 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/MasterData/WALs/jenkins-hbase4.apache.org,45369,1676214003194/jenkins-hbase4.apache.org%2C45369%2C1676214003194.1676214004962 2023-02-12 15:00:05,094 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46017,DS-308f6b74-90b2-4af6-b9b4-9f0dbec0aee8,DISK], DatanodeInfoWithStorage[127.0.0.1:39973,DS-e50680d0-d606-4c5b-bdce-8317dc34931c,DISK], DatanodeInfoWithStorage[127.0.0.1:33343,DS-cc807189-4c44-4066-9b81-0a7d2acae137,DISK]] 2023-02-12 15:00:05,095 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-12 15:00:05,095 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:05,099 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-12 15:00:05,100 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-12 15:00:05,168 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-12 15:00:05,175 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-12 15:00:05,198 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-12 15:00:05,210 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:05,216 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-12 15:00:05,218 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-12 15:00:05,236 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-12 15:00:05,241 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:05,242 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68965350, jitterRate=0.02766379714012146}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:05,242 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676214005137Initializing all the Stores at 1676214005139 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676214005140 (+1 ms)Cleaning up temporary data from old regions at 1676214005226 (+86 ms)Cleaning up detritus from prior splits at 1676214005227 (+1 ms)Region opened successfully at 1676214005242 (+15 ms) 2023-02-12 15:00:05,243 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-12 15:00:05,264 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-12 15:00:05,264 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-12 15:00:05,266 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-12 15:00:05,268 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-12 15:00:05,300 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-02-12 15:00:05,300 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-12 15:00:05,324 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-12 15:00:05,330 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-12 15:00:05,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-12 15:00:05,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-12 15:00:05,358 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-12 15:00:05,363 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-12 15:00:05,367 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-12 15:00:05,369 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 15:00:05,371 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-12 15:00:05,372 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-12 15:00:05,383 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-12 15:00:05,387 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-12 15:00:05,387 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-12 15:00:05,387 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-12 15:00:05,387 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-12 15:00:05,388 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 15:00:05,390 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,45369,1676214003194, sessionid=0x100490bcc700000, setting cluster-up flag (Was=false) 2023-02-12 15:00:05,407 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 15:00:05,414 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-12 15:00:05,415 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45369,1676214003194 2023-02-12 15:00:05,420 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 15:00:05,426 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-12 15:00:05,427 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45369,1676214003194 2023-02-12 15:00:05,429 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp 2023-02-12 15:00:05,486 INFO [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(952): ClusterId : 30424d8b-ad56-47a0-8406-ccf64d20b7a4 2023-02-12 15:00:05,486 INFO [RS:2;jenkins-hbase4:38453] regionserver.HRegionServer(952): ClusterId : 30424d8b-ad56-47a0-8406-ccf64d20b7a4 2023-02-12 15:00:05,486 INFO [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(952): ClusterId : 30424d8b-ad56-47a0-8406-ccf64d20b7a4 2023-02-12 15:00:05,493 DEBUG [RS:1;jenkins-hbase4:33845] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-12 15:00:05,493 DEBUG [RS:2;jenkins-hbase4:38453] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-12 15:00:05,493 DEBUG [RS:0;jenkins-hbase4:37113] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-12 15:00:05,499 DEBUG [RS:0;jenkins-hbase4:37113] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-12 15:00:05,499 DEBUG [RS:2;jenkins-hbase4:38453] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-12 15:00:05,499 DEBUG [RS:1;jenkins-hbase4:33845] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-12 15:00:05,500 DEBUG [RS:2;jenkins-hbase4:38453] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-12 15:00:05,500 DEBUG [RS:0;jenkins-hbase4:37113] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-12 15:00:05,500 DEBUG [RS:1;jenkins-hbase4:33845] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-12 15:00:05,505 DEBUG [RS:2;jenkins-hbase4:38453] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-12 15:00:05,505 DEBUG [RS:0;jenkins-hbase4:37113] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-12 15:00:05,505 DEBUG [RS:1;jenkins-hbase4:33845] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-12 15:00:05,507 DEBUG [RS:2;jenkins-hbase4:38453] zookeeper.ReadOnlyZKClient(139): Connect 0x5ef5b80c to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:05,507 DEBUG [RS:0;jenkins-hbase4:37113] zookeeper.ReadOnlyZKClient(139): Connect 0x5361090a to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:05,507 DEBUG [RS:1;jenkins-hbase4:33845] zookeeper.ReadOnlyZKClient(139): Connect 0x3abc1df0 to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:05,521 DEBUG [RS:2;jenkins-hbase4:38453] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50324da2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:05,522 DEBUG [RS:2;jenkins-hbase4:38453] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@540dd37c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-12 15:00:05,522 DEBUG [RS:0;jenkins-hbase4:37113] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5da6ed16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:05,522 DEBUG [RS:0;jenkins-hbase4:37113] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5999627a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-12 15:00:05,525 DEBUG [RS:1;jenkins-hbase4:33845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@347be4d7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:05,526 DEBUG [RS:1;jenkins-hbase4:33845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30447680, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-12 15:00:05,550 DEBUG [RS:0;jenkins-hbase4:37113] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:37113 2023-02-12 15:00:05,550 DEBUG [RS:1;jenkins-hbase4:33845] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:33845 2023-02-12 15:00:05,551 DEBUG [RS:2;jenkins-hbase4:38453] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:38453 2023-02-12 15:00:05,554 INFO [RS:1;jenkins-hbase4:33845] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-12 15:00:05,554 INFO [RS:2;jenkins-hbase4:38453] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-12 15:00:05,554 INFO [RS:0;jenkins-hbase4:37113] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-12 15:00:05,555 INFO [RS:0;jenkins-hbase4:37113] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-12 15:00:05,555 INFO [RS:2;jenkins-hbase4:38453] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-12 15:00:05,555 INFO [RS:1;jenkins-hbase4:33845] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-12 15:00:05,555 DEBUG [RS:2;jenkins-hbase4:38453] regionserver.HRegionServer(1023): About to register with Master. 2023-02-12 15:00:05,555 DEBUG [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(1023): About to register with Master. 2023-02-12 15:00:05,555 DEBUG [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(1023): About to register with Master. 2023-02-12 15:00:05,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-12 15:00:05,558 INFO [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45369,1676214003194 with isa=jenkins-hbase4.apache.org/172.31.14.131:37113, startcode=1676214004377 2023-02-12 15:00:05,558 INFO [RS:2;jenkins-hbase4:38453] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45369,1676214003194 with isa=jenkins-hbase4.apache.org/172.31.14.131:38453, startcode=1676214004461 2023-02-12 15:00:05,558 INFO [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45369,1676214003194 with isa=jenkins-hbase4.apache.org/172.31.14.131:33845, startcode=1676214004423 2023-02-12 15:00:05,567 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-12 15:00:05,567 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-12 15:00:05,567 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-12 15:00:05,567 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-12 15:00:05,567 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-12 15:00:05,567 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,567 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-12 15:00:05,567 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,572 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676214035572 2023-02-12 15:00:05,574 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-12 15:00:05,578 DEBUG [RS:1;jenkins-hbase4:33845] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-12 15:00:05,578 DEBUG [RS:0;jenkins-hbase4:37113] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-12 15:00:05,578 DEBUG [RS:2;jenkins-hbase4:38453] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-12 15:00:05,580 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-12 15:00:05,580 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-12 15:00:05,586 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-12 15:00:05,587 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-12 15:00:05,596 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-12 15:00:05,597 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-12 15:00:05,597 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-12 15:00:05,598 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-12 15:00:05,604 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,606 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-12 15:00:05,609 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-12 15:00:05,609 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-12 15:00:05,613 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-12 15:00:05,614 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-12 15:00:05,619 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676214005616,5,FailOnTimeoutGroup] 2023-02-12 15:00:05,621 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676214005620,5,FailOnTimeoutGroup] 2023-02-12 15:00:05,621 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,622 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-12 15:00:05,623 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,624 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,646 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-12 15:00:05,648 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-12 15:00:05,648 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b 2023-02-12 15:00:05,651 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36537, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-12 15:00:05,651 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42535, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-12 15:00:05,651 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34465, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-12 15:00:05,675 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:05,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45369] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:05,676 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45369] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:05,679 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:05,681 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-12 15:00:05,684 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/info 2023-02-12 15:00:05,685 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-12 15:00:05,686 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:05,686 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-12 15:00:05,689 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/rep_barrier 2023-02-12 15:00:05,690 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-12 15:00:05,691 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:05,691 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-12 15:00:05,694 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/table 2023-02-12 15:00:05,694 DEBUG [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b 2023-02-12 15:00:05,694 DEBUG [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b 2023-02-12 15:00:05,694 DEBUG [RS:2;jenkins-hbase4:38453] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b 2023-02-12 15:00:05,695 DEBUG [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35339 2023-02-12 15:00:05,695 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-12 15:00:05,695 DEBUG [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35339 2023-02-12 15:00:05,695 DEBUG [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-12 15:00:05,695 DEBUG [RS:2;jenkins-hbase4:38453] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35339 2023-02-12 15:00:05,695 DEBUG [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-12 15:00:05,696 DEBUG [RS:2;jenkins-hbase4:38453] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-12 15:00:05,696 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:05,698 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740 2023-02-12 15:00:05,698 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740 2023-02-12 15:00:05,702 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-12 15:00:05,704 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-12 15:00:05,705 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-12 15:00:05,706 DEBUG [RS:1;jenkins-hbase4:33845] zookeeper.ZKUtil(162): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:05,706 WARN [RS:1;jenkins-hbase4:33845] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-12 15:00:05,706 DEBUG [RS:0;jenkins-hbase4:37113] zookeeper.ZKUtil(162): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:05,706 INFO [RS:1;jenkins-hbase4:33845] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-12 15:00:05,706 DEBUG [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/WALs/jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:05,706 WARN [RS:0;jenkins-hbase4:37113] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-12 15:00:05,708 DEBUG [RS:2;jenkins-hbase4:38453] zookeeper.ZKUtil(162): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:05,709 WARN [RS:2;jenkins-hbase4:38453] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-12 15:00:05,708 INFO [RS:0;jenkins-hbase4:37113] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-12 15:00:05,709 INFO [RS:2;jenkins-hbase4:38453] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-12 15:00:05,709 DEBUG [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/WALs/jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:05,709 DEBUG [RS:2;jenkins-hbase4:38453] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/WALs/jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:05,712 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:05,712 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73923046, jitterRate=0.10153922438621521}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-12 15:00:05,713 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676214005679Initializing all the Stores at 1676214005680 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676214005680Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676214005681 (+1 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676214005681Cleaning up temporary data from old regions at 1676214005700 (+19 ms)Cleaning up detritus from prior splits at 1676214005701 (+1 ms)Region opened successfully at 1676214005713 (+12 ms) 2023-02-12 15:00:05,713 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-12 15:00:05,713 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-12 15:00:05,713 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-12 15:00:05,713 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-12 15:00:05,713 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-12 15:00:05,730 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-12 15:00:05,730 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676214005713Disabling compacts and flushes for region at 1676214005713Disabling writes for close at 1676214005713Writing region close event to WAL at 1676214005730 (+17 ms)Closed at 1676214005730 2023-02-12 15:00:05,730 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33845,1676214004423] 2023-02-12 15:00:05,731 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,38453,1676214004461] 2023-02-12 15:00:05,731 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,37113,1676214004377] 2023-02-12 15:00:05,739 DEBUG [RS:0;jenkins-hbase4:37113] zookeeper.ZKUtil(162): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:05,739 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-12 15:00:05,739 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-12 15:00:05,739 DEBUG [RS:2;jenkins-hbase4:38453] zookeeper.ZKUtil(162): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:05,740 DEBUG [RS:0;jenkins-hbase4:37113] zookeeper.ZKUtil(162): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:05,740 DEBUG [RS:1;jenkins-hbase4:33845] zookeeper.ZKUtil(162): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:05,740 DEBUG [RS:2;jenkins-hbase4:38453] zookeeper.ZKUtil(162): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:05,740 DEBUG [RS:0;jenkins-hbase4:37113] zookeeper.ZKUtil(162): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:05,741 DEBUG [RS:1;jenkins-hbase4:33845] zookeeper.ZKUtil(162): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:05,741 DEBUG [RS:2;jenkins-hbase4:38453] zookeeper.ZKUtil(162): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:05,742 DEBUG [RS:1;jenkins-hbase4:33845] zookeeper.ZKUtil(162): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:05,748 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-12 15:00:05,754 DEBUG [RS:0;jenkins-hbase4:37113] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-12 15:00:05,754 DEBUG [RS:2;jenkins-hbase4:38453] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-12 15:00:05,754 DEBUG [RS:1;jenkins-hbase4:33845] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-12 15:00:05,763 INFO [RS:0;jenkins-hbase4:37113] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-12 15:00:05,764 INFO [RS:2;jenkins-hbase4:38453] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-12 15:00:05,763 INFO [RS:1;jenkins-hbase4:33845] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-12 15:00:05,766 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-12 15:00:05,771 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:05,785 INFO [RS:1;jenkins-hbase4:33845] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-12 15:00:05,785 INFO [RS:0;jenkins-hbase4:37113] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-12 15:00:05,785 INFO [RS:2;jenkins-hbase4:38453] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-12 15:00:05,790 INFO [RS:2;jenkins-hbase4:38453] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-12 15:00:05,790 INFO [RS:0;jenkins-hbase4:37113] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-12 15:00:05,790 INFO [RS:1;jenkins-hbase4:33845] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-12 15:00:05,791 INFO [RS:0;jenkins-hbase4:37113] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,790 INFO [RS:2;jenkins-hbase4:38453] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,791 INFO [RS:1;jenkins-hbase4:33845] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,792 INFO [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-12 15:00:05,792 INFO [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-12 15:00:05,792 INFO [RS:2;jenkins-hbase4:38453] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-12 15:00:05,800 INFO [RS:2;jenkins-hbase4:38453] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,800 INFO [RS:0;jenkins-hbase4:37113] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,800 INFO [RS:1;jenkins-hbase4:33845] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,800 DEBUG [RS:2;jenkins-hbase4:38453] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,800 DEBUG [RS:0;jenkins-hbase4:37113] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,801 DEBUG [RS:1;jenkins-hbase4:33845] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,801 DEBUG [RS:0;jenkins-hbase4:37113] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,801 DEBUG [RS:1;jenkins-hbase4:33845] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,801 DEBUG [RS:0;jenkins-hbase4:37113] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,801 DEBUG [RS:1;jenkins-hbase4:33845] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,801 DEBUG [RS:0;jenkins-hbase4:37113] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,801 DEBUG [RS:1;jenkins-hbase4:33845] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,801 DEBUG [RS:0;jenkins-hbase4:37113] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,801 DEBUG [RS:1;jenkins-hbase4:33845] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,801 DEBUG [RS:0;jenkins-hbase4:37113] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-12 15:00:05,801 DEBUG [RS:1;jenkins-hbase4:33845] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-12 15:00:05,801 DEBUG [RS:0;jenkins-hbase4:37113] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,802 DEBUG [RS:1;jenkins-hbase4:33845] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,801 DEBUG [RS:2;jenkins-hbase4:38453] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,802 DEBUG [RS:1;jenkins-hbase4:33845] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,802 DEBUG [RS:2;jenkins-hbase4:38453] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,802 DEBUG [RS:1;jenkins-hbase4:33845] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,802 DEBUG [RS:2;jenkins-hbase4:38453] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,802 DEBUG [RS:1;jenkins-hbase4:33845] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,802 DEBUG [RS:2;jenkins-hbase4:38453] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,802 DEBUG [RS:0;jenkins-hbase4:37113] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,802 DEBUG [RS:2;jenkins-hbase4:38453] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-12 15:00:05,803 DEBUG [RS:0;jenkins-hbase4:37113] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,803 DEBUG [RS:2;jenkins-hbase4:38453] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,803 DEBUG [RS:0;jenkins-hbase4:37113] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,803 INFO [RS:1;jenkins-hbase4:33845] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,803 DEBUG [RS:2;jenkins-hbase4:38453] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,803 INFO [RS:1;jenkins-hbase4:33845] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,803 DEBUG [RS:2;jenkins-hbase4:38453] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,803 INFO [RS:1;jenkins-hbase4:33845] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,803 DEBUG [RS:2;jenkins-hbase4:38453] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-12 15:00:05,805 INFO [RS:0;jenkins-hbase4:37113] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,805 INFO [RS:2;jenkins-hbase4:38453] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,805 INFO [RS:0;jenkins-hbase4:37113] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,806 INFO [RS:2;jenkins-hbase4:38453] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,806 INFO [RS:0;jenkins-hbase4:37113] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,806 INFO [RS:2;jenkins-hbase4:38453] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,822 INFO [RS:0;jenkins-hbase4:37113] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-12 15:00:05,822 INFO [RS:1;jenkins-hbase4:33845] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-12 15:00:05,824 INFO [RS:2;jenkins-hbase4:38453] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-12 15:00:05,825 INFO [RS:0;jenkins-hbase4:37113] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37113,1676214004377-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,825 INFO [RS:2;jenkins-hbase4:38453] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38453,1676214004461-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,825 INFO [RS:1;jenkins-hbase4:33845] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33845,1676214004423-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:05,848 INFO [RS:1;jenkins-hbase4:33845] regionserver.Replication(203): jenkins-hbase4.apache.org,33845,1676214004423 started 2023-02-12 15:00:05,848 INFO [RS:0;jenkins-hbase4:37113] regionserver.Replication(203): jenkins-hbase4.apache.org,37113,1676214004377 started 2023-02-12 15:00:05,848 INFO [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,33845,1676214004423, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33845, sessionid=0x100490bcc700002 2023-02-12 15:00:05,848 INFO [RS:2;jenkins-hbase4:38453] regionserver.Replication(203): jenkins-hbase4.apache.org,38453,1676214004461 started 2023-02-12 15:00:05,848 INFO [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,37113,1676214004377, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:37113, sessionid=0x100490bcc700001 2023-02-12 15:00:05,848 INFO [RS:2;jenkins-hbase4:38453] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,38453,1676214004461, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:38453, sessionid=0x100490bcc700003 2023-02-12 15:00:05,848 DEBUG [RS:1;jenkins-hbase4:33845] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-12 15:00:05,849 DEBUG [RS:2;jenkins-hbase4:38453] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-12 15:00:05,848 DEBUG [RS:0;jenkins-hbase4:37113] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-12 15:00:05,849 DEBUG [RS:2;jenkins-hbase4:38453] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:05,849 DEBUG [RS:1;jenkins-hbase4:33845] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:05,849 DEBUG [RS:2;jenkins-hbase4:38453] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38453,1676214004461' 2023-02-12 15:00:05,849 DEBUG [RS:0;jenkins-hbase4:37113] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:05,850 DEBUG [RS:2;jenkins-hbase4:38453] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-12 15:00:05,850 DEBUG [RS:1;jenkins-hbase4:33845] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33845,1676214004423' 2023-02-12 15:00:05,850 DEBUG [RS:1;jenkins-hbase4:33845] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-12 15:00:05,850 DEBUG [RS:0;jenkins-hbase4:37113] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37113,1676214004377' 2023-02-12 15:00:05,850 DEBUG [RS:0;jenkins-hbase4:37113] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-12 15:00:05,851 DEBUG [RS:2;jenkins-hbase4:38453] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-12 15:00:05,851 DEBUG [RS:0;jenkins-hbase4:37113] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-12 15:00:05,851 DEBUG [RS:1;jenkins-hbase4:33845] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-12 15:00:05,851 DEBUG [RS:2;jenkins-hbase4:38453] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-12 15:00:05,851 DEBUG [RS:0;jenkins-hbase4:37113] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-12 15:00:05,852 DEBUG [RS:2;jenkins-hbase4:38453] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-12 15:00:05,852 DEBUG [RS:1;jenkins-hbase4:33845] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-12 15:00:05,852 DEBUG [RS:0;jenkins-hbase4:37113] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-12 15:00:05,852 DEBUG [RS:1;jenkins-hbase4:33845] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-12 15:00:05,852 DEBUG [RS:1;jenkins-hbase4:33845] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:05,852 DEBUG [RS:2;jenkins-hbase4:38453] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:05,852 DEBUG [RS:1;jenkins-hbase4:33845] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33845,1676214004423' 2023-02-12 15:00:05,852 DEBUG [RS:0;jenkins-hbase4:37113] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:05,853 DEBUG [RS:0;jenkins-hbase4:37113] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37113,1676214004377' 2023-02-12 15:00:05,853 DEBUG [RS:0;jenkins-hbase4:37113] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-12 15:00:05,853 DEBUG [RS:1;jenkins-hbase4:33845] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-12 15:00:05,853 DEBUG [RS:2;jenkins-hbase4:38453] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38453,1676214004461' 2023-02-12 15:00:05,853 DEBUG [RS:2;jenkins-hbase4:38453] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-12 15:00:05,853 DEBUG [RS:0;jenkins-hbase4:37113] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-12 15:00:05,853 DEBUG [RS:1;jenkins-hbase4:33845] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-12 15:00:05,853 DEBUG [RS:2;jenkins-hbase4:38453] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-12 15:00:05,854 DEBUG [RS:0;jenkins-hbase4:37113] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-12 15:00:05,854 INFO [RS:0;jenkins-hbase4:37113] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-12 15:00:05,854 INFO [RS:0;jenkins-hbase4:37113] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-12 15:00:05,854 DEBUG [RS:2;jenkins-hbase4:38453] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-12 15:00:05,854 DEBUG [RS:1;jenkins-hbase4:33845] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-12 15:00:05,854 INFO [RS:2;jenkins-hbase4:38453] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-12 15:00:05,855 INFO [RS:1;jenkins-hbase4:33845] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-12 15:00:05,855 INFO [RS:2;jenkins-hbase4:38453] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-12 15:00:05,856 INFO [RS:1;jenkins-hbase4:33845] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-12 15:00:05,923 DEBUG [jenkins-hbase4:45369] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-12 15:00:05,927 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-12 15:00:05,933 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 15:00:05,933 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 15:00:05,933 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 15:00:05,933 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 15:00:05,936 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37113,1676214004377, state=OPENING 2023-02-12 15:00:05,944 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-12 15:00:05,947 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 15:00:05,947 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-12 15:00:05,951 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:05,966 INFO [RS:0;jenkins-hbase4:37113] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37113%2C1676214004377, suffix=, logDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/WALs/jenkins-hbase4.apache.org,37113,1676214004377, archiveDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/oldWALs, maxLogs=32 2023-02-12 15:00:05,968 INFO [RS:2;jenkins-hbase4:38453] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38453%2C1676214004461, suffix=, logDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/WALs/jenkins-hbase4.apache.org,38453,1676214004461, archiveDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/oldWALs, maxLogs=32 2023-02-12 15:00:05,998 INFO [RS:1;jenkins-hbase4:33845] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33845%2C1676214004423, suffix=, logDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/WALs/jenkins-hbase4.apache.org,33845,1676214004423, archiveDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/oldWALs, maxLogs=32 2023-02-12 15:00:06,016 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39973,DS-e50680d0-d606-4c5b-bdce-8317dc34931c,DISK] 2023-02-12 15:00:06,018 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33343,DS-cc807189-4c44-4066-9b81-0a7d2acae137,DISK] 2023-02-12 15:00:06,018 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46017,DS-308f6b74-90b2-4af6-b9b4-9f0dbec0aee8,DISK] 2023-02-12 15:00:06,026 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33343,DS-cc807189-4c44-4066-9b81-0a7d2acae137,DISK] 2023-02-12 15:00:06,026 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39973,DS-e50680d0-d606-4c5b-bdce-8317dc34931c,DISK] 2023-02-12 15:00:06,026 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46017,DS-308f6b74-90b2-4af6-b9b4-9f0dbec0aee8,DISK] 2023-02-12 15:00:06,043 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46017,DS-308f6b74-90b2-4af6-b9b4-9f0dbec0aee8,DISK] 2023-02-12 15:00:06,043 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33343,DS-cc807189-4c44-4066-9b81-0a7d2acae137,DISK] 2023-02-12 15:00:06,043 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39973,DS-e50680d0-d606-4c5b-bdce-8317dc34931c,DISK] 2023-02-12 15:00:06,044 INFO [RS:0;jenkins-hbase4:37113] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/WALs/jenkins-hbase4.apache.org,37113,1676214004377/jenkins-hbase4.apache.org%2C37113%2C1676214004377.1676214005970 2023-02-12 15:00:06,044 INFO [RS:2;jenkins-hbase4:38453] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/WALs/jenkins-hbase4.apache.org,38453,1676214004461/jenkins-hbase4.apache.org%2C38453%2C1676214004461.1676214005970 2023-02-12 15:00:06,045 DEBUG [RS:2;jenkins-hbase4:38453] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33343,DS-cc807189-4c44-4066-9b81-0a7d2acae137,DISK], DatanodeInfoWithStorage[127.0.0.1:46017,DS-308f6b74-90b2-4af6-b9b4-9f0dbec0aee8,DISK], DatanodeInfoWithStorage[127.0.0.1:39973,DS-e50680d0-d606-4c5b-bdce-8317dc34931c,DISK]] 2023-02-12 15:00:06,046 DEBUG [RS:0;jenkins-hbase4:37113] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33343,DS-cc807189-4c44-4066-9b81-0a7d2acae137,DISK], DatanodeInfoWithStorage[127.0.0.1:46017,DS-308f6b74-90b2-4af6-b9b4-9f0dbec0aee8,DISK], DatanodeInfoWithStorage[127.0.0.1:39973,DS-e50680d0-d606-4c5b-bdce-8317dc34931c,DISK]] 2023-02-12 15:00:06,052 INFO [RS:1;jenkins-hbase4:33845] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/WALs/jenkins-hbase4.apache.org,33845,1676214004423/jenkins-hbase4.apache.org%2C33845%2C1676214004423.1676214006000 2023-02-12 15:00:06,052 DEBUG [RS:1;jenkins-hbase4:33845] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33343,DS-cc807189-4c44-4066-9b81-0a7d2acae137,DISK], DatanodeInfoWithStorage[127.0.0.1:39973,DS-e50680d0-d606-4c5b-bdce-8317dc34931c,DISK], DatanodeInfoWithStorage[127.0.0.1:46017,DS-308f6b74-90b2-4af6-b9b4-9f0dbec0aee8,DISK]] 2023-02-12 15:00:06,136 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:06,138 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-12 15:00:06,142 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56774, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-12 15:00:06,156 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-12 15:00:06,156 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-12 15:00:06,160 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37113%2C1676214004377.meta, suffix=.meta, logDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/WALs/jenkins-hbase4.apache.org,37113,1676214004377, archiveDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/oldWALs, maxLogs=32 2023-02-12 15:00:06,184 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33343,DS-cc807189-4c44-4066-9b81-0a7d2acae137,DISK] 2023-02-12 15:00:06,184 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39973,DS-e50680d0-d606-4c5b-bdce-8317dc34931c,DISK] 2023-02-12 15:00:06,186 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46017,DS-308f6b74-90b2-4af6-b9b4-9f0dbec0aee8,DISK] 2023-02-12 15:00:06,192 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/WALs/jenkins-hbase4.apache.org,37113,1676214004377/jenkins-hbase4.apache.org%2C37113%2C1676214004377.meta.1676214006162.meta 2023-02-12 15:00:06,192 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33343,DS-cc807189-4c44-4066-9b81-0a7d2acae137,DISK], DatanodeInfoWithStorage[127.0.0.1:39973,DS-e50680d0-d606-4c5b-bdce-8317dc34931c,DISK], DatanodeInfoWithStorage[127.0.0.1:46017,DS-308f6b74-90b2-4af6-b9b4-9f0dbec0aee8,DISK]] 2023-02-12 15:00:06,192 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-12 15:00:06,194 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-12 15:00:06,211 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-12 15:00:06,216 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-12 15:00:06,222 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-12 15:00:06,222 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:06,222 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-12 15:00:06,222 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-12 15:00:06,225 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-12 15:00:06,227 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/info 2023-02-12 15:00:06,227 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/info 2023-02-12 15:00:06,228 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-12 15:00:06,229 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:06,229 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-12 15:00:06,230 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/rep_barrier 2023-02-12 15:00:06,231 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/rep_barrier 2023-02-12 15:00:06,231 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-12 15:00:06,232 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:06,232 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-12 15:00:06,233 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/table 2023-02-12 15:00:06,234 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/table 2023-02-12 15:00:06,234 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-12 15:00:06,235 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:06,237 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740 2023-02-12 15:00:06,240 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740 2023-02-12 15:00:06,244 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-12 15:00:06,247 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-12 15:00:06,249 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72748531, jitterRate=0.08403758704662323}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-12 15:00:06,249 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676214006222Writing region info on filesystem at 1676214006222Initializing all the Stores at 1676214006224 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676214006224Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676214006224Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676214006225 (+1 ms)Cleaning up temporary data from old regions at 1676214006242 (+17 ms)Cleaning up detritus from prior splits at 1676214006243 (+1 ms)Running coprocessor post-open hooks at 1676214006249 (+6 ms)Region opened successfully at 1676214006249 2023-02-12 15:00:06,259 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676214006128 2023-02-12 15:00:06,278 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-12 15:00:06,280 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-12 15:00:06,280 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37113,1676214004377, state=OPEN 2023-02-12 15:00:06,283 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-12 15:00:06,283 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-12 15:00:06,288 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-12 15:00:06,288 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37113,1676214004377 in 332 msec 2023-02-12 15:00:06,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-12 15:00:06,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 541 msec 2023-02-12 15:00:06,300 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 825 msec 2023-02-12 15:00:06,300 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676214004508, completionTime=-1 2023-02-12 15:00:06,300 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-12 15:00:06,300 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-12 15:00:06,364 DEBUG [hconnection-0x779b93d1-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:06,367 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56790, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:06,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-12 15:00:06,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676214066387 2023-02-12 15:00:06,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676214126387 2023-02-12 15:00:06,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 87 msec 2023-02-12 15:00:06,415 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45369,1676214003194-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:06,416 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45369,1676214003194-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:06,416 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45369,1676214003194-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:06,417 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:45369, period=300000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:06,418 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-12 15:00:06,429 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-12 15:00:06,437 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-12 15:00:06,438 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-12 15:00:06,448 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-12 15:00:06,450 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-12 15:00:06,453 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-12 15:00:06,477 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/hbase/namespace/2e9d615367e89c5413b0d2c088c828a2 2023-02-12 15:00:06,480 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/hbase/namespace/2e9d615367e89c5413b0d2c088c828a2 empty. 2023-02-12 15:00:06,481 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/hbase/namespace/2e9d615367e89c5413b0d2c088c828a2 2023-02-12 15:00:06,481 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-12 15:00:06,524 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-12 15:00:06,527 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 2e9d615367e89c5413b0d2c088c828a2, NAME => 'hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:06,547 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:06,547 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 2e9d615367e89c5413b0d2c088c828a2, disabling compactions & flushes 2023-02-12 15:00:06,547 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. 2023-02-12 15:00:06,547 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. 2023-02-12 15:00:06,547 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. after waiting 0 ms 2023-02-12 15:00:06,547 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. 2023-02-12 15:00:06,547 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. 2023-02-12 15:00:06,547 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 2e9d615367e89c5413b0d2c088c828a2: Waiting for close lock at 1676214006547Disabling compacts and flushes for region at 1676214006547Disabling writes for close at 1676214006547Writing region close event to WAL at 1676214006547Closed at 1676214006547 2023-02-12 15:00:06,552 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-12 15:00:06,568 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676214006555"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214006555"}]},"ts":"1676214006555"} 2023-02-12 15:00:06,597 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-12 15:00:06,599 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-12 15:00:06,603 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214006599"}]},"ts":"1676214006599"} 2023-02-12 15:00:06,608 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-12 15:00:06,613 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-12 15:00:06,614 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 15:00:06,614 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 15:00:06,614 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 15:00:06,614 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 15:00:06,617 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=2e9d615367e89c5413b0d2c088c828a2, ASSIGN}] 2023-02-12 15:00:06,620 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=2e9d615367e89c5413b0d2c088c828a2, ASSIGN 2023-02-12 15:00:06,622 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=2e9d615367e89c5413b0d2c088c828a2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=false 2023-02-12 15:00:06,774 INFO [jenkins-hbase4:45369] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-12 15:00:06,775 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=2e9d615367e89c5413b0d2c088c828a2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:06,775 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676214006775"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214006775"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214006775"}]},"ts":"1676214006775"} 2023-02-12 15:00:06,780 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 2e9d615367e89c5413b0d2c088c828a2, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:06,934 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:06,935 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-12 15:00:06,938 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47804, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-12 15:00:06,945 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. 2023-02-12 15:00:06,947 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 2e9d615367e89c5413b0d2c088c828a2, NAME => 'hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2.', STARTKEY => '', ENDKEY => ''} 2023-02-12 15:00:06,948 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 2e9d615367e89c5413b0d2c088c828a2 2023-02-12 15:00:06,948 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:06,948 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 2e9d615367e89c5413b0d2c088c828a2 2023-02-12 15:00:06,948 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 2e9d615367e89c5413b0d2c088c828a2 2023-02-12 15:00:06,950 INFO [StoreOpener-2e9d615367e89c5413b0d2c088c828a2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 2e9d615367e89c5413b0d2c088c828a2 2023-02-12 15:00:06,952 DEBUG [StoreOpener-2e9d615367e89c5413b0d2c088c828a2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/namespace/2e9d615367e89c5413b0d2c088c828a2/info 2023-02-12 15:00:06,952 DEBUG [StoreOpener-2e9d615367e89c5413b0d2c088c828a2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/namespace/2e9d615367e89c5413b0d2c088c828a2/info 2023-02-12 15:00:06,953 INFO [StoreOpener-2e9d615367e89c5413b0d2c088c828a2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2e9d615367e89c5413b0d2c088c828a2 columnFamilyName info 2023-02-12 15:00:06,953 INFO [StoreOpener-2e9d615367e89c5413b0d2c088c828a2-1] regionserver.HStore(310): Store=2e9d615367e89c5413b0d2c088c828a2/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:06,955 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/namespace/2e9d615367e89c5413b0d2c088c828a2 2023-02-12 15:00:06,956 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/namespace/2e9d615367e89c5413b0d2c088c828a2 2023-02-12 15:00:06,961 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 2e9d615367e89c5413b0d2c088c828a2 2023-02-12 15:00:06,965 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/namespace/2e9d615367e89c5413b0d2c088c828a2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:06,966 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 2e9d615367e89c5413b0d2c088c828a2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63521139, jitterRate=-0.05346126854419708}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:06,966 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 2e9d615367e89c5413b0d2c088c828a2: Running coprocessor pre-open hook at 1676214006948Writing region info on filesystem at 1676214006948Initializing all the Stores at 1676214006950 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676214006950Cleaning up temporary data from old regions at 1676214006957 (+7 ms)Cleaning up detritus from prior splits at 1676214006958 (+1 ms)Running coprocessor post-open hooks at 1676214006966 (+8 ms)Region opened successfully at 1676214006966 2023-02-12 15:00:06,968 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2., pid=6, masterSystemTime=1676214006934 2023-02-12 15:00:06,972 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. 2023-02-12 15:00:06,973 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. 2023-02-12 15:00:06,974 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=2e9d615367e89c5413b0d2c088c828a2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:06,975 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676214006973"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214006973"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214006973"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214006973"}]},"ts":"1676214006973"} 2023-02-12 15:00:06,982 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-12 15:00:06,982 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 2e9d615367e89c5413b0d2c088c828a2, server=jenkins-hbase4.apache.org,33845,1676214004423 in 198 msec 2023-02-12 15:00:06,986 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-12 15:00:06,987 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=2e9d615367e89c5413b0d2c088c828a2, ASSIGN in 365 msec 2023-02-12 15:00:06,988 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-12 15:00:06,989 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214006988"}]},"ts":"1676214006988"} 2023-02-12 15:00:06,991 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-12 15:00:06,994 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-12 15:00:06,997 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 554 msec 2023-02-12 15:00:07,051 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-12 15:00:07,054 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-12 15:00:07,054 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 15:00:07,076 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:07,079 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47816, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:07,096 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-12 15:00:07,118 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-12 15:00:07,124 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 38 msec 2023-02-12 15:00:07,129 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-12 15:00:07,141 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-12 15:00:07,146 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-02-12 15:00:07,154 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-12 15:00:07,158 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-12 15:00:07,161 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.641sec 2023-02-12 15:00:07,163 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-12 15:00:07,165 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-12 15:00:07,165 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-12 15:00:07,166 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45369,1676214003194-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-12 15:00:07,168 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45369,1676214003194-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-12 15:00:07,213 DEBUG [Listener at localhost/38323] zookeeper.ReadOnlyZKClient(139): Connect 0x1c633018 to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:07,215 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-12 15:00:07,221 DEBUG [Listener at localhost/38323] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b210801, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:07,234 DEBUG [hconnection-0x7b9b1635-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:07,244 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56796, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:07,253 INFO [Listener at localhost/38323] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,45369,1676214003194 2023-02-12 15:00:07,282 INFO [Listener at localhost/38323] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=361, OpenFileDescriptor=695, MaxFileDescriptor=60000, SystemLoadAverage=211, ProcessCount=171, AvailableMemoryMB=3907 2023-02-12 15:00:07,295 DEBUG [Listener at localhost/38323] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-12 15:00:07,299 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37720, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-12 15:00:07,314 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testRestoreSnapshot-1676214007285', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-12 15:00:07,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:07,319 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676214007285 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-12 15:00:07,322 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676214007285 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-12 15:00:07,324 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676214007285" procId is: 9 2023-02-12 15:00:07,331 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:07,331 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:07,331 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:07,331 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:07,332 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 empty. 2023-02-12 15:00:07,332 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:07,332 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:07,333 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 empty. 2023-02-12 15:00:07,333 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe empty. 2023-02-12 15:00:07,333 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c empty. 2023-02-12 15:00:07,335 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:07,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-12 15:00:07,336 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 empty. 2023-02-12 15:00:07,336 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:07,336 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f empty. 2023-02-12 15:00:07,336 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:07,337 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:07,337 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:07,340 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:07,340 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676214007285 regions 2023-02-12 15:00:07,362 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/.tabledesc/.tableinfo.0000000001 2023-02-12 15:00:07,364 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(7671): creating {ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676214007285', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:07,365 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(7671): creating {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676214007285', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:07,365 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(7671): creating {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676214007285', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:07,408 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:07,408 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1603): Closing c70c08e0263941ae1bfbb2d57f1c3052, disabling compactions & flushes 2023-02-12 15:00:07,408 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:07,409 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:07,409 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. after waiting 0 ms 2023-02-12 15:00:07,410 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:07,410 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:07,410 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1557): Region close journal for c70c08e0263941ae1bfbb2d57f1c3052: Waiting for close lock at 1676214007408Disabling compacts and flushes for region at 1676214007408Disabling writes for close at 1676214007410 (+2 ms)Writing region close event to WAL at 1676214007410Closed at 1676214007410 2023-02-12 15:00:07,411 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(7671): creating {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676214007285', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:07,414 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:07,414 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1603): Closing 469de65a1b57113b3e7565d5f5bf8201, disabling compactions & flushes 2023-02-12 15:00:07,414 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:07,414 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:07,414 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:07,415 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1603): Closing e31ecbd7855a2e0bca06cbe75ecfc8fe, disabling compactions & flushes 2023-02-12 15:00:07,415 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. after waiting 0 ms 2023-02-12 15:00:07,415 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:07,415 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:07,415 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:07,416 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:07,416 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. after waiting 0 ms 2023-02-12 15:00:07,416 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1557): Region close journal for 469de65a1b57113b3e7565d5f5bf8201: Waiting for close lock at 1676214007414Disabling compacts and flushes for region at 1676214007414Disabling writes for close at 1676214007415 (+1 ms)Writing region close event to WAL at 1676214007415Closed at 1676214007416 (+1 ms) 2023-02-12 15:00:07,416 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:07,416 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:07,416 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1557): Region close journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Waiting for close lock at 1676214007415Disabling compacts and flushes for region at 1676214007415Disabling writes for close at 1676214007416 (+1 ms)Writing region close event to WAL at 1676214007416Closed at 1676214007416 2023-02-12 15:00:07,417 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(7671): creating {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676214007285', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:07,417 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(7671): creating {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676214007285', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:07,441 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:07,443 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1603): Closing 87a15dc17c09c51160706b761e764ba2, disabling compactions & flushes 2023-02-12 15:00:07,443 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:07,443 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:07,443 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. after waiting 0 ms 2023-02-12 15:00:07,443 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:07,443 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:07,443 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-2] regionserver.HRegion(1557): Region close journal for 87a15dc17c09c51160706b761e764ba2: Waiting for close lock at 1676214007443Disabling compacts and flushes for region at 1676214007443Disabling writes for close at 1676214007443Writing region close event to WAL at 1676214007443Closed at 1676214007443 2023-02-12 15:00:07,448 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:07,449 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1603): Closing e116768b6cb1e8fb57dfa7c68914d39c, disabling compactions & flushes 2023-02-12 15:00:07,449 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:07,449 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:07,449 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. after waiting 0 ms 2023-02-12 15:00:07,449 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:07,449 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:07,449 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-1] regionserver.HRegion(1557): Region close journal for e116768b6cb1e8fb57dfa7c68914d39c: Waiting for close lock at 1676214007449Disabling compacts and flushes for region at 1676214007449Disabling writes for close at 1676214007449Writing region close event to WAL at 1676214007449Closed at 1676214007449 2023-02-12 15:00:07,450 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:07,450 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1603): Closing a2b878baabc37b3076d648a696d2d33f, disabling compactions & flushes 2023-02-12 15:00:07,450 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:07,450 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:07,450 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. after waiting 0 ms 2023-02-12 15:00:07,451 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:07,451 INFO [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:07,451 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676214007285-pool-0] regionserver.HRegion(1557): Region close journal for a2b878baabc37b3076d648a696d2d33f: Waiting for close lock at 1676214007450Disabling compacts and flushes for region at 1676214007450Disabling writes for close at 1676214007451 (+1 ms)Writing region close event to WAL at 1676214007451Closed at 1676214007451 2023-02-12 15:00:07,456 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676214007285 execute state=CREATE_TABLE_ADD_TO_META 2023-02-12 15:00:07,458 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214007457"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214007457"}]},"ts":"1676214007457"} 2023-02-12 15:00:07,458 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214007457"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214007457"}]},"ts":"1676214007457"} 2023-02-12 15:00:07,458 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214007457"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214007457"}]},"ts":"1676214007457"} 2023-02-12 15:00:07,459 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214007457"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214007457"}]},"ts":"1676214007457"} 2023-02-12 15:00:07,459 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214007457"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214007457"}]},"ts":"1676214007457"} 2023-02-12 15:00:07,459 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214007457"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214007457"}]},"ts":"1676214007457"} 2023-02-12 15:00:07,506 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-12 15:00:07,507 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676214007285 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-12 15:00:07,507 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214007507"}]},"ts":"1676214007507"} 2023-02-12 15:00:07,510 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLING in hbase:meta 2023-02-12 15:00:07,514 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-12 15:00:07,519 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 15:00:07,519 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 15:00:07,519 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 15:00:07,519 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 15:00:07,520 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN}] 2023-02-12 15:00:07,524 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN 2023-02-12 15:00:07,524 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN 2023-02-12 15:00:07,525 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN 2023-02-12 15:00:07,525 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN 2023-02-12 15:00:07,526 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=false 2023-02-12 15:00:07,527 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=false 2023-02-12 15:00:07,527 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=false 2023-02-12 15:00:07,527 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=false 2023-02-12 15:00:07,527 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN 2023-02-12 15:00:07,528 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN 2023-02-12 15:00:07,529 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=false 2023-02-12 15:00:07,529 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=false 2023-02-12 15:00:07,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-12 15:00:07,676 INFO [jenkins-hbase4:45369] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 15:00:07,679 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:07,679 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:07,679 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:07,679 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214007679"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214007679"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214007679"}]},"ts":"1676214007679"} 2023-02-12 15:00:07,679 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:07,680 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214007679"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214007679"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214007679"}]},"ts":"1676214007679"} 2023-02-12 15:00:07,680 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214007679"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214007679"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214007679"}]},"ts":"1676214007679"} 2023-02-12 15:00:07,679 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:07,679 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214007679"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214007679"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214007679"}]},"ts":"1676214007679"} 2023-02-12 15:00:07,680 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214007679"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214007679"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214007679"}]},"ts":"1676214007679"} 2023-02-12 15:00:07,683 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=11, state=RUNNABLE; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:07,687 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=13, state=RUNNABLE; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:07,688 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=12, state=RUNNABLE; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:07,692 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=14, state=RUNNABLE; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:07,694 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=15, state=RUNNABLE; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:07,695 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:07,695 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214007694"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214007694"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214007694"}]},"ts":"1676214007694"} 2023-02-12 15:00:07,699 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=10, state=RUNNABLE; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:07,841 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:07,841 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-12 15:00:07,843 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60758, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-12 15:00:07,846 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:07,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 15:00:07,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:07,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:07,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:07,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:07,850 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:07,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 15:00:07,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:07,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:07,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:07,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:07,851 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:07,853 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:07,853 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:07,853 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:07,854 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:07,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 15:00:07,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:07,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:07,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:07,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:07,856 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:07,856 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:07,857 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:07,858 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 87a15dc17c09c51160706b761e764ba2 columnFamilyName cf 2023-02-12 15:00:07,858 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a2b878baabc37b3076d648a696d2d33f columnFamilyName cf 2023-02-12 15:00:07,859 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:07,859 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:07,860 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e116768b6cb1e8fb57dfa7c68914d39c columnFamilyName cf 2023-02-12 15:00:07,862 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(310): Store=a2b878baabc37b3076d648a696d2d33f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:07,862 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(310): Store=87a15dc17c09c51160706b761e764ba2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:07,862 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(310): Store=e116768b6cb1e8fb57dfa7c68914d39c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:07,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:07,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:07,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:07,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:07,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:07,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:07,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:07,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:07,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:07,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:07,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:07,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:07,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e116768b6cb1e8fb57dfa7c68914d39c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67307626, jitterRate=0.002961784601211548}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:07,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a2b878baabc37b3076d648a696d2d33f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66385377, jitterRate=-0.01078079640865326}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:07,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 87a15dc17c09c51160706b761e764ba2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61768997, jitterRate=-0.0795702189207077}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:07,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a2b878baabc37b3076d648a696d2d33f: Running coprocessor pre-open hook at 1676214007847Writing region info on filesystem at 1676214007847Initializing all the Stores at 1676214007849 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214007850 (+1 ms)Cleaning up temporary data from old regions at 1676214007869 (+19 ms)Cleaning up detritus from prior splits at 1676214007871 (+2 ms)Running coprocessor post-open hooks at 1676214007890 (+19 ms)Region opened successfully at 1676214007890 2023-02-12 15:00:07,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e116768b6cb1e8fb57dfa7c68914d39c: Running coprocessor pre-open hook at 1676214007855Writing region info on filesystem at 1676214007855Initializing all the Stores at 1676214007856 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214007856Cleaning up temporary data from old regions at 1676214007870 (+14 ms)Cleaning up detritus from prior splits at 1676214007871 (+1 ms)Running coprocessor post-open hooks at 1676214007889 (+18 ms)Region opened successfully at 1676214007889 2023-02-12 15:00:07,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 87a15dc17c09c51160706b761e764ba2: Running coprocessor pre-open hook at 1676214007851Writing region info on filesystem at 1676214007851Initializing all the Stores at 1676214007852 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214007853 (+1 ms)Cleaning up temporary data from old regions at 1676214007871 (+18 ms)Cleaning up detritus from prior splits at 1676214007871Running coprocessor post-open hooks at 1676214007890 (+19 ms)Region opened successfully at 1676214007890 2023-02-12 15:00:07,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c., pid=20, masterSystemTime=1676214007848 2023-02-12 15:00:07,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f., pid=19, masterSystemTime=1676214007839 2023-02-12 15:00:07,893 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2., pid=17, masterSystemTime=1676214007841 2023-02-12 15:00:07,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:07,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:07,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:07,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 15:00:07,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:07,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:07,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:07,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:07,898 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:07,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:07,899 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:07,899 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214007898"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214007898"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214007898"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214007898"}]},"ts":"1676214007898"} 2023-02-12 15:00:07,899 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:07,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 15:00:07,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:07,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:07,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:07,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:07,900 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:07,901 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214007900"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214007900"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214007900"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214007900"}]},"ts":"1676214007900"} 2023-02-12 15:00:07,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:07,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:07,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:07,904 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:07,904 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:07,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 15:00:07,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:07,905 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214007902"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214007902"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214007902"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214007902"}]},"ts":"1676214007902"} 2023-02-12 15:00:07,906 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:07,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:07,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:07,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:07,909 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:07,909 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:07,909 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:07,910 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:07,910 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e31ecbd7855a2e0bca06cbe75ecfc8fe columnFamilyName cf 2023-02-12 15:00:07,910 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:07,910 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 469de65a1b57113b3e7565d5f5bf8201 columnFamilyName cf 2023-02-12 15:00:07,912 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(310): Store=469de65a1b57113b3e7565d5f5bf8201/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:07,913 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(310): Store=e31ecbd7855a2e0bca06cbe75ecfc8fe/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:07,914 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=14 2023-02-12 15:00:07,914 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=14, state=SUCCESS; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,37113,1676214004377 in 213 msec 2023-02-12 15:00:07,916 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:07,916 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:07,916 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:07,916 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c70c08e0263941ae1bfbb2d57f1c3052 columnFamilyName cf 2023-02-12 15:00:07,917 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:07,917 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:07,919 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:07,919 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=15 2023-02-12 15:00:07,919 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=15, state=SUCCESS; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423 in 214 msec 2023-02-12 15:00:07,920 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(310): Store=c70c08e0263941ae1bfbb2d57f1c3052/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:07,922 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN in 394 msec 2023-02-12 15:00:07,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:07,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:07,923 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=13 2023-02-12 15:00:07,924 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=13, state=SUCCESS; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,38453,1676214004461 in 226 msec 2023-02-12 15:00:07,925 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:07,925 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN in 399 msec 2023-02-12 15:00:07,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN in 404 msec 2023-02-12 15:00:07,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:07,929 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 469de65a1b57113b3e7565d5f5bf8201; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69445741, jitterRate=0.03482218086719513}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:07,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 469de65a1b57113b3e7565d5f5bf8201: Running coprocessor pre-open hook at 1676214007900Writing region info on filesystem at 1676214007900Initializing all the Stores at 1676214007905 (+5 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214007905Cleaning up temporary data from old regions at 1676214007918 (+13 ms)Cleaning up detritus from prior splits at 1676214007919 (+1 ms)Running coprocessor post-open hooks at 1676214007929 (+10 ms)Region opened successfully at 1676214007929 2023-02-12 15:00:07,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:07,930 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201., pid=21, masterSystemTime=1676214007848 2023-02-12 15:00:07,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:07,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:07,938 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:07,938 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214007937"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214007937"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214007937"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214007937"}]},"ts":"1676214007937"} 2023-02-12 15:00:07,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:07,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c70c08e0263941ae1bfbb2d57f1c3052; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71078558, jitterRate=0.05915305018424988}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:07,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c70c08e0263941ae1bfbb2d57f1c3052: Running coprocessor pre-open hook at 1676214007907Writing region info on filesystem at 1676214007907Initializing all the Stores at 1676214007909 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214007909Cleaning up temporary data from old regions at 1676214007925 (+16 ms)Cleaning up detritus from prior splits at 1676214007926 (+1 ms)Running coprocessor post-open hooks at 1676214007939 (+13 ms)Region opened successfully at 1676214007939 2023-02-12 15:00:07,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:07,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052., pid=18, masterSystemTime=1676214007841 2023-02-12 15:00:07,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:07,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:07,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:07,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e31ecbd7855a2e0bca06cbe75ecfc8fe; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61117679, jitterRate=-0.08927561342716217}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:07,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Running coprocessor pre-open hook at 1676214007898Writing region info on filesystem at 1676214007898Initializing all the Stores at 1676214007901 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214007901Cleaning up temporary data from old regions at 1676214007922 (+21 ms)Cleaning up detritus from prior splits at 1676214007935 (+13 ms)Running coprocessor post-open hooks at 1676214007946 (+11 ms)Region opened successfully at 1676214007946 2023-02-12 15:00:07,947 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:07,947 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214007946"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214007946"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214007946"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214007946"}]},"ts":"1676214007946"} 2023-02-12 15:00:07,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe., pid=16, masterSystemTime=1676214007839 2023-02-12 15:00:07,949 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=10 2023-02-12 15:00:07,949 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=10, state=SUCCESS; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,33845,1676214004423 in 244 msec 2023-02-12 15:00:07,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:07,952 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:07,952 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN in 429 msec 2023-02-12 15:00:07,953 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:07,953 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214007953"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214007953"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214007953"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214007953"}]},"ts":"1676214007953"} 2023-02-12 15:00:07,956 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=12 2023-02-12 15:00:07,956 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=12, state=SUCCESS; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461 in 264 msec 2023-02-12 15:00:07,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN in 436 msec 2023-02-12 15:00:07,961 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=11 2023-02-12 15:00:07,961 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=11, state=SUCCESS; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377 in 273 msec 2023-02-12 15:00:07,965 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=9 2023-02-12 15:00:07,965 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN in 441 msec 2023-02-12 15:00:07,966 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676214007285 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-12 15:00:07,966 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214007966"}]},"ts":"1676214007966"} 2023-02-12 15:00:07,968 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLED in hbase:meta 2023-02-12 15:00:07,973 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676214007285 execute state=CREATE_TABLE_POST_OPERATION 2023-02-12 15:00:07,976 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676214007285 in 658 msec 2023-02-12 15:00:08,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-12 15:00:08,093 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 9 completed 2023-02-12 15:00:08,100 DEBUG [Listener at localhost/38323] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676214007285 2023-02-12 15:00:08,102 DEBUG [Listener at localhost/38323] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:08,105 INFO [Listener at localhost/38323] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-12 15:00:08,125 DEBUG [hconnection-0x7b9b1635-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:08,128 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47824, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:08,132 DEBUG [hconnection-0x7b9b1635-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:08,135 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60768, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:08,146 INFO [Listener at localhost/38323] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676214007285 2023-02-12 15:00:08,150 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676214007285 2023-02-12 15:00:08,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:08,161 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214008161"}]},"ts":"1676214008161"} 2023-02-12 15:00:08,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-12 15:00:08,163 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLING in hbase:meta 2023-02-12 15:00:08,165 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676214007285 to state=DISABLING 2023-02-12 15:00:08,168 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN}] 2023-02-12 15:00:08,171 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN 2023-02-12 15:00:08,171 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN 2023-02-12 15:00:08,171 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN 2023-02-12 15:00:08,171 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN 2023-02-12 15:00:08,172 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN 2023-02-12 15:00:08,173 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:08,173 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:08,173 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214008173"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214008173"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214008173"}]},"ts":"1676214008173"} 2023-02-12 15:00:08,174 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214008173"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214008173"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214008173"}]},"ts":"1676214008173"} 2023-02-12 15:00:08,174 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:08,173 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:08,174 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214008174"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214008174"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214008174"}]},"ts":"1676214008174"} 2023-02-12 15:00:08,174 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214008173"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214008173"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214008173"}]},"ts":"1676214008173"} 2023-02-12 15:00:08,174 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:08,175 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214008174"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214008174"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214008174"}]},"ts":"1676214008174"} 2023-02-12 15:00:08,177 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=24, state=RUNNABLE; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:08,181 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=23, state=RUNNABLE; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:08,183 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=27, state=RUNNABLE; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:08,185 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=28, state=RUNNABLE; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:08,188 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN 2023-02-12 15:00:08,188 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=26, state=RUNNABLE; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:08,191 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:08,191 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214008191"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214008191"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214008191"}]},"ts":"1676214008191"} 2023-02-12 15:00:08,194 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=25, state=RUNNABLE; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:08,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:08,339 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:08,339 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e116768b6cb1e8fb57dfa7c68914d39c, disabling compactions & flushes 2023-02-12 15:00:08,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a2b878baabc37b3076d648a696d2d33f, disabling compactions & flushes 2023-02-12 15:00:08,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:08,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:08,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:08,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:08,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. after waiting 0 ms 2023-02-12 15:00:08,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. after waiting 0 ms 2023-02-12 15:00:08,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:08,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:08,345 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:08,345 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c70c08e0263941ae1bfbb2d57f1c3052, disabling compactions & flushes 2023-02-12 15:00:08,346 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:08,346 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:08,346 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. after waiting 0 ms 2023-02-12 15:00:08,346 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:08,350 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 15:00:08,350 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 15:00:08,351 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:08,351 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:08,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a2b878baabc37b3076d648a696d2d33f: Waiting for close lock at 1676214008339Running coprocessor pre-close hooks at 1676214008339Disabling compacts and flushes for region at 1676214008339Disabling writes for close at 1676214008340 (+1 ms)Writing region close event to WAL at 1676214008341 (+1 ms)Running coprocessor post-close hooks at 1676214008351 (+10 ms)Closed at 1676214008351 2023-02-12 15:00:08,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e116768b6cb1e8fb57dfa7c68914d39c: Waiting for close lock at 1676214008339Running coprocessor pre-close hooks at 1676214008339Disabling compacts and flushes for region at 1676214008339Disabling writes for close at 1676214008340 (+1 ms)Writing region close event to WAL at 1676214008341 (+1 ms)Running coprocessor post-close hooks at 1676214008351 (+10 ms)Closed at 1676214008351 2023-02-12 15:00:08,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 15:00:08,355 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSED 2023-02-12 15:00:08,356 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214008355"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214008355"}]},"ts":"1676214008355"} 2023-02-12 15:00:08,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:08,357 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSED 2023-02-12 15:00:08,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:08,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:08,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:08,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 469de65a1b57113b3e7565d5f5bf8201, disabling compactions & flushes 2023-02-12 15:00:08,359 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:08,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:08,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. after waiting 0 ms 2023-02-12 15:00:08,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:08,357 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214008357"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214008357"}]},"ts":"1676214008357"} 2023-02-12 15:00:08,358 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e31ecbd7855a2e0bca06cbe75ecfc8fe, disabling compactions & flushes 2023-02-12 15:00:08,359 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:08,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:08,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. after waiting 0 ms 2023-02-12 15:00:08,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:08,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:08,367 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c70c08e0263941ae1bfbb2d57f1c3052: Waiting for close lock at 1676214008345Running coprocessor pre-close hooks at 1676214008345Disabling compacts and flushes for region at 1676214008345Disabling writes for close at 1676214008346 (+1 ms)Writing region close event to WAL at 1676214008346Running coprocessor post-close hooks at 1676214008358 (+12 ms)Closed at 1676214008365 (+7 ms) 2023-02-12 15:00:08,372 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:08,372 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:08,372 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=28 2023-02-12 15:00:08,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 87a15dc17c09c51160706b761e764ba2, disabling compactions & flushes 2023-02-12 15:00:08,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:08,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:08,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. after waiting 0 ms 2023-02-12 15:00:08,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:08,373 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=28, state=SUCCESS; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423 in 175 msec 2023-02-12 15:00:08,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 15:00:08,374 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=27 2023-02-12 15:00:08,375 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 15:00:08,374 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSED 2023-02-12 15:00:08,375 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=27, state=SUCCESS; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,37113,1676214004377 in 185 msec 2023-02-12 15:00:08,375 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214008374"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214008374"}]},"ts":"1676214008374"} 2023-02-12 15:00:08,375 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:08,376 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 469de65a1b57113b3e7565d5f5bf8201: Waiting for close lock at 1676214008359Running coprocessor pre-close hooks at 1676214008359Disabling compacts and flushes for region at 1676214008359Disabling writes for close at 1676214008359Writing region close event to WAL at 1676214008359Running coprocessor post-close hooks at 1676214008375 (+16 ms)Closed at 1676214008375 2023-02-12 15:00:08,378 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:08,378 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN in 205 msec 2023-02-12 15:00:08,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Waiting for close lock at 1676214008358Running coprocessor pre-close hooks at 1676214008358Disabling compacts and flushes for region at 1676214008358Disabling writes for close at 1676214008360 (+2 ms)Writing region close event to WAL at 1676214008365 (+5 ms)Running coprocessor post-close hooks at 1676214008377 (+12 ms)Closed at 1676214008378 (+1 ms) 2023-02-12 15:00:08,378 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:08,379 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN in 207 msec 2023-02-12 15:00:08,380 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSED 2023-02-12 15:00:08,380 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214008380"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214008380"}]},"ts":"1676214008380"} 2023-02-12 15:00:08,381 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:08,382 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSED 2023-02-12 15:00:08,383 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214008382"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214008382"}]},"ts":"1676214008382"} 2023-02-12 15:00:08,383 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=25 2023-02-12 15:00:08,383 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=25, state=SUCCESS; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461 in 184 msec 2023-02-12 15:00:08,386 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN in 216 msec 2023-02-12 15:00:08,388 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=23 2023-02-12 15:00:08,388 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=23, state=SUCCESS; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,33845,1676214004423 in 202 msec 2023-02-12 15:00:08,390 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=24 2023-02-12 15:00:08,390 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=24, state=SUCCESS; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377 in 208 msec 2023-02-12 15:00:08,391 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN in 221 msec 2023-02-12 15:00:08,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 15:00:08,393 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN in 223 msec 2023-02-12 15:00:08,393 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:08,393 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 87a15dc17c09c51160706b761e764ba2: Waiting for close lock at 1676214008372Running coprocessor pre-close hooks at 1676214008372Disabling compacts and flushes for region at 1676214008372Disabling writes for close at 1676214008373 (+1 ms)Writing region close event to WAL at 1676214008374 (+1 ms)Running coprocessor post-close hooks at 1676214008393 (+19 ms)Closed at 1676214008393 2023-02-12 15:00:08,395 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:08,395 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSED 2023-02-12 15:00:08,396 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214008395"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214008395"}]},"ts":"1676214008395"} 2023-02-12 15:00:08,400 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=26 2023-02-12 15:00:08,400 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=26, state=SUCCESS; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,38453,1676214004461 in 209 msec 2023-02-12 15:00:08,403 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=22 2023-02-12 15:00:08,403 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN in 233 msec 2023-02-12 15:00:08,404 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214008404"}]},"ts":"1676214008404"} 2023-02-12 15:00:08,406 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLED in hbase:meta 2023-02-12 15:00:08,408 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676214007285 to state=DISABLED 2023-02-12 15:00:08,412 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676214007285 in 258 msec 2023-02-12 15:00:08,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-12 15:00:08,414 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 22 completed 2023-02-12 15:00:08,429 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:08,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676214008430 (current time:1676214008430). 2023-02-12 15:00:08,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 15:00:08,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(139): Connect 0x473faa0c to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:08,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ae6daba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:08,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:08,445 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56804, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:08,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x473faa0c to 127.0.0.1:61893 2023-02-12 15:00:08,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:08,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 15:00:08,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:08,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:08,474 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:08,476 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE; timeout=600000, timestamp=1676214608476 2023-02-12 15:00:08,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:08,476 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676214007285 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676214007285 2023-02-12 15:00:08,482 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 15:00:08,483 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 15:00:08,501 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } 2023-02-12 15:00:08,504 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:08,504 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:08,508 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:08,508 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:08,508 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:08,509 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:08,509 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:08,509 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:08,509 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:08,509 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:08,509 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:08,509 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:08,509 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:08,509 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:08,517 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:08,517 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:08,517 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:08,519 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:08,519 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:08,531 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:08,532 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:08,733 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 15:00:08,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } is done 2023-02-12 15:00:08,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }' is still in progress! 2023-02-12 15:00:08,738 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-12 15:00:09,002 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } as finished. 2023-02-12 15:00:09,002 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676214007285 2023-02-12 15:00:09,005 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/emptySnaptb-1676214007285 2023-02-12 15:00:09,049 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/emptySnaptb-1676214007285 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/emptySnaptb-1676214007285 2023-02-12 15:00:09,079 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676214007285 of table testRestoreSnapshot-1676214007285 completed 2023-02-12 15:00:09,079 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/emptySnaptb-1676214007285 2023-02-12 15:00:09,080 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/emptySnaptb-1676214007285 2023-02-12 15:00:09,080 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676214007285 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676214007285 at 1676214008477Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } at 1676214008501 (+24 ms)Consolidate snapshot: emptySnaptb-1676214007285 at 1676214009002 (+501 ms)Loading Region manifests for emptySnaptb-1676214007285 at 1676214009003 (+1 ms)Writing data manifest for emptySnaptb-1676214007285 at 1676214009018 (+15 ms)Verifying snapshot: emptySnaptb-1676214007285 at 1676214009036 (+18 ms)Snapshot emptySnaptb-1676214007285 of table testRestoreSnapshot-1676214007285 completed at 1676214009079 (+43 ms) 2023-02-12 15:00:09,081 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:09,083 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE in 617 msec 2023-02-12 15:00:09,238 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 15:00:09,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } is done 2023-02-12 15:00:09,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 15:00:09,241 INFO [Listener at localhost/38323] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676214007285 2023-02-12 15:00:09,245 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676214007285 2023-02-12 15:00:09,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:09,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-12 15:00:09,258 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676214007285 2023-02-12 15:00:09,258 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214009258"}]},"ts":"1676214009258"} 2023-02-12 15:00:09,260 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLING in hbase:meta 2023-02-12 15:00:09,263 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 15:00:09,266 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN}] 2023-02-12 15:00:09,268 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN 2023-02-12 15:00:09,269 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN 2023-02-12 15:00:09,269 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN 2023-02-12 15:00:09,269 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN 2023-02-12 15:00:09,270 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN 2023-02-12 15:00:09,270 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:09,270 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:09,270 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=true 2023-02-12 15:00:09,270 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=true 2023-02-12 15:00:09,270 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:09,271 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=true 2023-02-12 15:00:09,270 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:09,271 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=true 2023-02-12 15:00:09,272 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:09,272 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=true 2023-02-12 15:00:09,273 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN 2023-02-12 15:00:09,274 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:09,274 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=true 2023-02-12 15:00:09,421 INFO [jenkins-hbase4:45369] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 15:00:09,421 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:09,421 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:09,421 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:09,421 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:09,422 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214009421"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214009421"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214009421"}]},"ts":"1676214009421"} 2023-02-12 15:00:09,421 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:09,422 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214009421"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214009421"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214009421"}]},"ts":"1676214009421"} 2023-02-12 15:00:09,422 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214009421"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214009421"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214009421"}]},"ts":"1676214009421"} 2023-02-12 15:00:09,422 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214009421"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214009421"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214009421"}]},"ts":"1676214009421"} 2023-02-12 15:00:09,422 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214009421"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214009421"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214009421"}]},"ts":"1676214009421"} 2023-02-12 15:00:09,424 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=41, state=RUNNABLE; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:09,425 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=40, state=RUNNABLE; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:09,426 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=39, state=RUNNABLE; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:09,428 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=42, state=RUNNABLE; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:09,429 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:09,429 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214009428"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214009428"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214009428"}]},"ts":"1676214009428"} 2023-02-12 15:00:09,429 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=38, state=RUNNABLE; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:09,431 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=37, state=RUNNABLE; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:09,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-12 15:00:09,581 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:09,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 15:00:09,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:09,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:09,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:09,581 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:09,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:09,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 15:00:09,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:09,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:09,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:09,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:09,583 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:09,584 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:09,585 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:09,585 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:09,585 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e31ecbd7855a2e0bca06cbe75ecfc8fe columnFamilyName cf 2023-02-12 15:00:09,585 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:09,586 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:09,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 15:00:09,586 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:09,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:09,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:09,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:09,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:09,587 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(310): Store=e31ecbd7855a2e0bca06cbe75ecfc8fe/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:09,587 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c70c08e0263941ae1bfbb2d57f1c3052 columnFamilyName cf 2023-02-12 15:00:09,588 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(310): Store=c70c08e0263941ae1bfbb2d57f1c3052/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:09,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:09,588 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:09,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:09,590 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:09,590 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:09,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:09,591 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e116768b6cb1e8fb57dfa7c68914d39c columnFamilyName cf 2023-02-12 15:00:09,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:09,592 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(310): Store=e116768b6cb1e8fb57dfa7c68914d39c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:09,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:09,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:09,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:09,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:09,596 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e31ecbd7855a2e0bca06cbe75ecfc8fe; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64207710, jitterRate=-0.0432305634021759}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:09,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Running coprocessor pre-open hook at 1676214009582Writing region info on filesystem at 1676214009582Initializing all the Stores at 1676214009583 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214009583Cleaning up temporary data from old regions at 1676214009591 (+8 ms)Cleaning up detritus from prior splits at 1676214009591Running coprocessor post-open hooks at 1676214009596 (+5 ms)Region opened successfully at 1676214009596 2023-02-12 15:00:09,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c70c08e0263941ae1bfbb2d57f1c3052; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70422292, jitterRate=0.04937392473220825}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:09,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c70c08e0263941ae1bfbb2d57f1c3052: Running coprocessor pre-open hook at 1676214009582Writing region info on filesystem at 1676214009582Initializing all the Stores at 1676214009583 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214009584 (+1 ms)Cleaning up temporary data from old regions at 1676214009592 (+8 ms)Cleaning up detritus from prior splits at 1676214009593 (+1 ms)Running coprocessor post-open hooks at 1676214009597 (+4 ms)Region opened successfully at 1676214009598 (+1 ms) 2023-02-12 15:00:09,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe., pid=47, masterSystemTime=1676214009577 2023-02-12 15:00:09,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052., pid=45, masterSystemTime=1676214009578 2023-02-12 15:00:09,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:09,601 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:09,601 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:09,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 15:00:09,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:09,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:09,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:09,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:09,602 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:09,602 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:09,602 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214009602"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214009602"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214009602"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214009602"}]},"ts":"1676214009602"} 2023-02-12 15:00:09,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:09,603 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:09,603 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:09,603 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e116768b6cb1e8fb57dfa7c68914d39c; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70508498, jitterRate=0.05065849423408508}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:09,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 15:00:09,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e116768b6cb1e8fb57dfa7c68914d39c: Running coprocessor pre-open hook at 1676214009586Writing region info on filesystem at 1676214009586Initializing all the Stores at 1676214009588 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214009588Cleaning up temporary data from old regions at 1676214009597 (+9 ms)Cleaning up detritus from prior splits at 1676214009598 (+1 ms)Running coprocessor post-open hooks at 1676214009603 (+5 ms)Region opened successfully at 1676214009603 2023-02-12 15:00:09,603 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:09,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:09,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:09,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:09,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:09,604 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:09,605 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c., pid=46, masterSystemTime=1676214009582 2023-02-12 15:00:09,605 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214009604"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214009604"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214009604"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214009604"}]},"ts":"1676214009604"} 2023-02-12 15:00:09,606 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:09,606 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:09,606 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:09,606 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a2b878baabc37b3076d648a696d2d33f columnFamilyName cf 2023-02-12 15:00:09,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:09,608 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:09,608 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:09,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 15:00:09,608 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:09,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:09,608 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214009608"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214009608"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214009608"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214009608"}]},"ts":"1676214009608"} 2023-02-12 15:00:09,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:09,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:09,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:09,609 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(310): Store=a2b878baabc37b3076d648a696d2d33f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:09,610 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:09,610 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:09,611 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=38 2023-02-12 15:00:09,611 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=38, state=SUCCESS; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377 in 178 msec 2023-02-12 15:00:09,611 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 87a15dc17c09c51160706b761e764ba2 columnFamilyName cf 2023-02-12 15:00:09,612 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:09,613 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(310): Store=87a15dc17c09c51160706b761e764ba2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:09,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:09,613 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=39 2023-02-12 15:00:09,614 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=39, state=SUCCESS; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461 in 182 msec 2023-02-12 15:00:09,614 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:09,614 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:09,614 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 469de65a1b57113b3e7565d5f5bf8201 columnFamilyName cf 2023-02-12 15:00:09,615 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:09,615 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN in 347 msec 2023-02-12 15:00:09,615 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:09,616 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN in 350 msec 2023-02-12 15:00:09,617 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:09,617 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(310): Store=469de65a1b57113b3e7565d5f5bf8201/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:09,617 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=42 2023-02-12 15:00:09,618 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=42, state=SUCCESS; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423 in 184 msec 2023-02-12 15:00:09,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:09,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:09,621 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN in 354 msec 2023-02-12 15:00:09,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:09,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:09,622 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a2b878baabc37b3076d648a696d2d33f; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62918700, jitterRate=-0.06243830919265747}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:09,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a2b878baabc37b3076d648a696d2d33f: Running coprocessor pre-open hook at 1676214009602Writing region info on filesystem at 1676214009602Initializing all the Stores at 1676214009603 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214009603Cleaning up temporary data from old regions at 1676214009616 (+13 ms)Cleaning up detritus from prior splits at 1676214009617 (+1 ms)Running coprocessor post-open hooks at 1676214009622 (+5 ms)Region opened successfully at 1676214009622 2023-02-12 15:00:09,623 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 87a15dc17c09c51160706b761e764ba2; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61307353, jitterRate=-0.08644925057888031}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:09,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 87a15dc17c09c51160706b761e764ba2: Running coprocessor pre-open hook at 1676214009604Writing region info on filesystem at 1676214009604Initializing all the Stores at 1676214009606 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214009606Cleaning up temporary data from old regions at 1676214009617 (+11 ms)Cleaning up detritus from prior splits at 1676214009618 (+1 ms)Running coprocessor post-open hooks at 1676214009623 (+5 ms)Region opened successfully at 1676214009623 2023-02-12 15:00:09,623 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f., pid=43, masterSystemTime=1676214009577 2023-02-12 15:00:09,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2., pid=44, masterSystemTime=1676214009578 2023-02-12 15:00:09,626 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:09,627 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:09,627 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:09,627 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214009626"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214009626"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214009626"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214009626"}]},"ts":"1676214009626"} 2023-02-12 15:00:09,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:09,627 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:09,628 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:09,629 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:09,629 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214009628"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214009628"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214009628"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214009628"}]},"ts":"1676214009628"} 2023-02-12 15:00:09,629 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 469de65a1b57113b3e7565d5f5bf8201; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66892825, jitterRate=-0.003219231963157654}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:09,629 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 469de65a1b57113b3e7565d5f5bf8201: Running coprocessor pre-open hook at 1676214009609Writing region info on filesystem at 1676214009609Initializing all the Stores at 1676214009611 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214009611Cleaning up temporary data from old regions at 1676214009622 (+11 ms)Cleaning up detritus from prior splits at 1676214009623 (+1 ms)Running coprocessor post-open hooks at 1676214009629 (+6 ms)Region opened successfully at 1676214009629 2023-02-12 15:00:09,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201., pid=48, masterSystemTime=1676214009582 2023-02-12 15:00:09,633 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=41 2023-02-12 15:00:09,633 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=41, state=SUCCESS; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,37113,1676214004377 in 206 msec 2023-02-12 15:00:09,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:09,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:09,635 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:09,635 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214009634"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214009634"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214009634"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214009634"}]},"ts":"1676214009634"} 2023-02-12 15:00:09,636 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=40 2023-02-12 15:00:09,636 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=40, state=SUCCESS; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,38453,1676214004461 in 206 msec 2023-02-12 15:00:09,637 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN in 369 msec 2023-02-12 15:00:09,638 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN in 373 msec 2023-02-12 15:00:09,639 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=37 2023-02-12 15:00:09,640 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=37, state=SUCCESS; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,33845,1676214004423 in 206 msec 2023-02-12 15:00:09,642 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-02-12 15:00:09,642 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN in 376 msec 2023-02-12 15:00:09,643 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214009643"}]},"ts":"1676214009643"} 2023-02-12 15:00:09,645 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLED in hbase:meta 2023-02-12 15:00:09,647 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676214007285' was successfully enabled. 2023-02-12 15:00:09,649 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676214007285 in 400 msec 2023-02-12 15:00:10,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-12 15:00:10,010 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 36 completed 2023-02-12 15:00:10,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:10,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:10,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:10,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:10,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:10,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:10,064 DEBUG [Listener at localhost/38323] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676214007285 2023-02-12 15:00:10,064 DEBUG [Listener at localhost/38323] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:10,066 INFO [Listener at localhost/38323] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-12 15:00:10,088 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 15:00:10,090 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 15:00:10,092 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 15:00:10,094 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 15:00:10,096 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 15:00:10,106 INFO [Listener at localhost/38323] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676214007285 2023-02-12 15:00:10,106 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676214007285 2023-02-12 15:00:10,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:10,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-12 15:00:10,112 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214010112"}]},"ts":"1676214010112"} 2023-02-12 15:00:10,114 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLING in hbase:meta 2023-02-12 15:00:10,116 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676214007285 to state=DISABLING 2023-02-12 15:00:10,117 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN}] 2023-02-12 15:00:10,119 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN 2023-02-12 15:00:10,119 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN 2023-02-12 15:00:10,119 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN 2023-02-12 15:00:10,119 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN 2023-02-12 15:00:10,120 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN 2023-02-12 15:00:10,120 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:10,120 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214010120"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214010120"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214010120"}]},"ts":"1676214010120"} 2023-02-12 15:00:10,121 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:10,121 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:10,121 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214010121"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214010121"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214010121"}]},"ts":"1676214010121"} 2023-02-12 15:00:10,121 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:10,121 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214010121"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214010121"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214010121"}]},"ts":"1676214010121"} 2023-02-12 15:00:10,121 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:10,121 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214010121"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214010121"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214010121"}]},"ts":"1676214010121"} 2023-02-12 15:00:10,121 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214010121"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214010121"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214010121"}]},"ts":"1676214010121"} 2023-02-12 15:00:10,122 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=52, state=RUNNABLE; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:10,124 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=53, state=RUNNABLE; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:10,126 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=50, state=RUNNABLE; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:10,127 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=51, state=RUNNABLE; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:10,128 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN 2023-02-12 15:00:10,128 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=55, state=RUNNABLE; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:10,130 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:10,130 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214010130"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214010130"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214010130"}]},"ts":"1676214010130"} 2023-02-12 15:00:10,132 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=54, state=RUNNABLE; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:10,276 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:10,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c70c08e0263941ae1bfbb2d57f1c3052, disabling compactions & flushes 2023-02-12 15:00:10,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:10,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:10,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. after waiting 0 ms 2023-02-12 15:00:10,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:10,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c70c08e0263941ae1bfbb2d57f1c3052 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-12 15:00:10,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:10,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:10,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 469de65a1b57113b3e7565d5f5bf8201, disabling compactions & flushes 2023-02-12 15:00:10,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e31ecbd7855a2e0bca06cbe75ecfc8fe, disabling compactions & flushes 2023-02-12 15:00:10,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:10,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:10,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:10,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:10,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. after waiting 0 ms 2023-02-12 15:00:10,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. after waiting 0 ms 2023-02-12 15:00:10,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:10,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:10,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 469de65a1b57113b3e7565d5f5bf8201 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-12 15:00:10,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e31ecbd7855a2e0bca06cbe75ecfc8fe 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-12 15:00:10,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-12 15:00:10,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b 2023-02-12 15:00:10,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c 2023-02-12 15:00:10,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 2023-02-12 15:00:10,441 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/.tmp/cf/498d9c0eb3304fc7aedc40e7676473e2 2023-02-12 15:00:10,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/.tmp/cf/4a211d423a9d40e08d63ef96097f6015 2023-02-12 15:00:10,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/.tmp/cf/e0edae35a0ff4d6db2a54417a26dc5a6 2023-02-12 15:00:10,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/.tmp/cf/498d9c0eb3304fc7aedc40e7676473e2 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2 2023-02-12 15:00:10,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/.tmp/cf/4a211d423a9d40e08d63ef96097f6015 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015 2023-02-12 15:00:10,455 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/.tmp/cf/e0edae35a0ff4d6db2a54417a26dc5a6 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6 2023-02-12 15:00:10,461 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2, entries=29, sequenceid=8, filesize=9.8 K 2023-02-12 15:00:10,461 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015, entries=39, sequenceid=8, filesize=11.5 K 2023-02-12 15:00:10,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for c70c08e0263941ae1bfbb2d57f1c3052 in 187ms, sequenceid=8, compaction requested=false 2023-02-12 15:00:10,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for e31ecbd7855a2e0bca06cbe75ecfc8fe in 183ms, sequenceid=8, compaction requested=false 2023-02-12 15:00:10,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6, entries=36, sequenceid=8, filesize=11.0 K 2023-02-12 15:00:10,466 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 469de65a1b57113b3e7565d5f5bf8201 in 184ms, sequenceid=8, compaction requested=false 2023-02-12 15:00:10,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676214007285' 2023-02-12 15:00:10,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676214007285' 2023-02-12 15:00:10,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676214007285' 2023-02-12 15:00:10,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 15:00:10,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 15:00:10,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 15:00:10,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:10,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:10,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Waiting for close lock at 1676214010280Running coprocessor pre-close hooks at 1676214010280Disabling compacts and flushes for region at 1676214010280Disabling writes for close at 1676214010282 (+2 ms)Obtaining lock to block concurrent updates at 1676214010282Preparing flush snapshotting stores in e31ecbd7855a2e0bca06cbe75ecfc8fe at 1676214010282Finished memstore snapshotting testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe., syncing WAL and waiting on mvcc, flushsize=dataSize=2611, getHeapSize=5856, getOffHeapSize=0, getCellsCount=39 at 1676214010291 (+9 ms)Flushing stores of testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. at 1676214010293 (+2 ms)Flushing e31ecbd7855a2e0bca06cbe75ecfc8fe/cf: creating writer at 1676214010297 (+4 ms)Flushing mob file e31ecbd7855a2e0bca06cbe75ecfc8fe/cf: appending metadata at 1676214010334 (+37 ms)Flushing mob file e31ecbd7855a2e0bca06cbe75ecfc8fe/cf: closing flushed file at 1676214010334Flushing e31ecbd7855a2e0bca06cbe75ecfc8fe/cf: appending metadata at 1676214010405 (+71 ms)Flushing e31ecbd7855a2e0bca06cbe75ecfc8fe/cf: closing flushed file at 1676214010409 (+4 ms)Flushing e31ecbd7855a2e0bca06cbe75ecfc8fe/cf: reopening flushed file at 1676214010454 (+45 ms)Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for e31ecbd7855a2e0bca06cbe75ecfc8fe in 183ms, sequenceid=8, compaction requested=false at 1676214010465 (+11 ms)Writing region close event to WAL at 1676214010474 (+9 ms)Running coprocessor post-close hooks at 1676214010479 (+5 ms)Closed at 1676214010479 2023-02-12 15:00:10,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 469de65a1b57113b3e7565d5f5bf8201: Waiting for close lock at 1676214010280Running coprocessor pre-close hooks at 1676214010280Disabling compacts and flushes for region at 1676214010280Disabling writes for close at 1676214010282 (+2 ms)Obtaining lock to block concurrent updates at 1676214010282Preparing flush snapshotting stores in 469de65a1b57113b3e7565d5f5bf8201 at 1676214010282Finished memstore snapshotting testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676214010291 (+9 ms)Flushing stores of testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. at 1676214010294 (+3 ms)Flushing 469de65a1b57113b3e7565d5f5bf8201/cf: creating writer at 1676214010297 (+3 ms)Flushing mob file 469de65a1b57113b3e7565d5f5bf8201/cf: appending metadata at 1676214010334 (+37 ms)Flushing mob file 469de65a1b57113b3e7565d5f5bf8201/cf: closing flushed file at 1676214010334Flushing 469de65a1b57113b3e7565d5f5bf8201/cf: appending metadata at 1676214010405 (+71 ms)Flushing 469de65a1b57113b3e7565d5f5bf8201/cf: closing flushed file at 1676214010409 (+4 ms)Flushing 469de65a1b57113b3e7565d5f5bf8201/cf: reopening flushed file at 1676214010456 (+47 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 469de65a1b57113b3e7565d5f5bf8201 in 184ms, sequenceid=8, compaction requested=false at 1676214010466 (+10 ms)Writing region close event to WAL at 1676214010474 (+8 ms)Running coprocessor post-close hooks at 1676214010479 (+5 ms)Closed at 1676214010479 2023-02-12 15:00:10,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:10,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c70c08e0263941ae1bfbb2d57f1c3052: Waiting for close lock at 1676214010276Running coprocessor pre-close hooks at 1676214010276Disabling compacts and flushes for region at 1676214010276Disabling writes for close at 1676214010277 (+1 ms)Obtaining lock to block concurrent updates at 1676214010279 (+2 ms)Preparing flush snapshotting stores in c70c08e0263941ae1bfbb2d57f1c3052 at 1676214010279Finished memstore snapshotting testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676214010291 (+12 ms)Flushing stores of testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. at 1676214010293 (+2 ms)Flushing c70c08e0263941ae1bfbb2d57f1c3052/cf: creating writer at 1676214010297 (+4 ms)Flushing mob file c70c08e0263941ae1bfbb2d57f1c3052/cf: appending metadata at 1676214010334 (+37 ms)Flushing mob file c70c08e0263941ae1bfbb2d57f1c3052/cf: closing flushed file at 1676214010334Flushing c70c08e0263941ae1bfbb2d57f1c3052/cf: appending metadata at 1676214010405 (+71 ms)Flushing c70c08e0263941ae1bfbb2d57f1c3052/cf: closing flushed file at 1676214010409 (+4 ms)Flushing c70c08e0263941ae1bfbb2d57f1c3052/cf: reopening flushed file at 1676214010454 (+45 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for c70c08e0263941ae1bfbb2d57f1c3052 in 187ms, sequenceid=8, compaction requested=false at 1676214010465 (+11 ms)Writing region close event to WAL at 1676214010474 (+9 ms)Running coprocessor post-close hooks at 1676214010479 (+5 ms)Closed at 1676214010479 2023-02-12 15:00:10,481 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:10,481 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:10,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a2b878baabc37b3076d648a696d2d33f, disabling compactions & flushes 2023-02-12 15:00:10,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:10,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:10,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. after waiting 0 ms 2023-02-12 15:00:10,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:10,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing a2b878baabc37b3076d648a696d2d33f 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-12 15:00:10,482 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSED 2023-02-12 15:00:10,482 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214010482"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214010482"}]},"ts":"1676214010482"} 2023-02-12 15:00:10,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:10,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:10,483 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e116768b6cb1e8fb57dfa7c68914d39c, disabling compactions & flushes 2023-02-12 15:00:10,483 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:10,483 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:10,483 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. after waiting 0 ms 2023-02-12 15:00:10,483 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:10,483 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e116768b6cb1e8fb57dfa7c68914d39c 1/1 column families, dataSize=21.59 KB heapSize=46.66 KB 2023-02-12 15:00:10,484 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSED 2023-02-12 15:00:10,484 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214010484"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214010484"}]},"ts":"1676214010484"} 2023-02-12 15:00:10,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:10,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:10,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 87a15dc17c09c51160706b761e764ba2, disabling compactions & flushes 2023-02-12 15:00:10,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:10,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:10,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. after waiting 0 ms 2023-02-12 15:00:10,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:10,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 87a15dc17c09c51160706b761e764ba2 1/1 column families, dataSize=2.29 KB heapSize=5.17 KB 2023-02-12 15:00:10,487 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSED 2023-02-12 15:00:10,487 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214010487"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214010487"}]},"ts":"1676214010487"} 2023-02-12 15:00:10,495 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=51 2023-02-12 15:00:10,495 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=51, state=SUCCESS; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377 in 359 msec 2023-02-12 15:00:10,499 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN in 378 msec 2023-02-12 15:00:10,499 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=50 2023-02-12 15:00:10,499 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=50, state=SUCCESS; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,33845,1676214004423 in 367 msec 2023-02-12 15:00:10,503 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=52 2023-02-12 15:00:10,504 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN in 382 msec 2023-02-12 15:00:10,504 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=52, state=SUCCESS; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461 in 375 msec 2023-02-12 15:00:10,506 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN in 387 msec 2023-02-12 15:00:10,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 2023-02-12 15:00:10,545 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 2023-02-12 15:00:10,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a 2023-02-12 15:00:10,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/.tmp/cf/f5c092a8a2e84ffe91bb687f3ea1c42d 2023-02-12 15:00:10,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/.tmp/cf/5718cdbb97f24b3081185dfde2fb1e55 2023-02-12 15:00:10,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=21.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/.tmp/cf/b00b668a6b014eb4a4e4c4bb861069f4 2023-02-12 15:00:10,574 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/.tmp/cf/f5c092a8a2e84ffe91bb687f3ea1c42d as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d 2023-02-12 15:00:10,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/.tmp/cf/5718cdbb97f24b3081185dfde2fb1e55 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55 2023-02-12 15:00:10,582 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d, entries=31, sequenceid=8, filesize=10.2 K 2023-02-12 15:00:10,583 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/.tmp/cf/b00b668a6b014eb4a4e4c4bb861069f4 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4 2023-02-12 15:00:10,583 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for a2b878baabc37b3076d648a696d2d33f in 101ms, sequenceid=8, compaction requested=false 2023-02-12 15:00:10,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55, entries=35, sequenceid=8, filesize=10.9 K 2023-02-12 15:00:10,588 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for 87a15dc17c09c51160706b761e764ba2 in 102ms, sequenceid=8, compaction requested=false 2023-02-12 15:00:10,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 15:00:10,593 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:10,593 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a2b878baabc37b3076d648a696d2d33f: Waiting for close lock at 1676214010481Running coprocessor pre-close hooks at 1676214010481Disabling compacts and flushes for region at 1676214010481Disabling writes for close at 1676214010482 (+1 ms)Obtaining lock to block concurrent updates at 1676214010482Preparing flush snapshotting stores in a2b878baabc37b3076d648a696d2d33f at 1676214010482Finished memstore snapshotting testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676214010482Flushing stores of testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. at 1676214010483 (+1 ms)Flushing a2b878baabc37b3076d648a696d2d33f/cf: creating writer at 1676214010483Flushing mob file a2b878baabc37b3076d648a696d2d33f/cf: appending metadata at 1676214010495 (+12 ms)Flushing mob file a2b878baabc37b3076d648a696d2d33f/cf: closing flushed file at 1676214010495Flushing a2b878baabc37b3076d648a696d2d33f/cf: appending metadata at 1676214010539 (+44 ms)Flushing a2b878baabc37b3076d648a696d2d33f/cf: closing flushed file at 1676214010539Flushing a2b878baabc37b3076d648a696d2d33f/cf: reopening flushed file at 1676214010575 (+36 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for a2b878baabc37b3076d648a696d2d33f in 101ms, sequenceid=8, compaction requested=false at 1676214010583 (+8 ms)Writing region close event to WAL at 1676214010586 (+3 ms)Running coprocessor post-close hooks at 1676214010593 (+7 ms)Closed at 1676214010593 2023-02-12 15:00:10,595 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4, entries=330, sequenceid=8, filesize=61.1 K 2023-02-12 15:00:10,596 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:10,596 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.59 KB/22106, heapSize ~46.64 KB/47760, currentSize=0 B/0 for e116768b6cb1e8fb57dfa7c68914d39c in 113ms, sequenceid=8, compaction requested=false 2023-02-12 15:00:10,597 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSED 2023-02-12 15:00:10,597 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214010597"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214010597"}]},"ts":"1676214010597"} 2023-02-12 15:00:10,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 15:00:10,601 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=54 2023-02-12 15:00:10,602 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=54, state=SUCCESS; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,37113,1676214004377 in 467 msec 2023-02-12 15:00:10,603 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:10,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 87a15dc17c09c51160706b761e764ba2: Waiting for close lock at 1676214010485Running coprocessor pre-close hooks at 1676214010485Disabling compacts and flushes for region at 1676214010485Disabling writes for close at 1676214010486 (+1 ms)Obtaining lock to block concurrent updates at 1676214010486Preparing flush snapshotting stores in 87a15dc17c09c51160706b761e764ba2 at 1676214010486Finished memstore snapshotting testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2., syncing WAL and waiting on mvcc, flushsize=dataSize=2343, getHeapSize=5280, getOffHeapSize=0, getCellsCount=35 at 1676214010486Flushing stores of testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. at 1676214010489 (+3 ms)Flushing 87a15dc17c09c51160706b761e764ba2/cf: creating writer at 1676214010492 (+3 ms)Flushing mob file 87a15dc17c09c51160706b761e764ba2/cf: appending metadata at 1676214010501 (+9 ms)Flushing mob file 87a15dc17c09c51160706b761e764ba2/cf: closing flushed file at 1676214010502 (+1 ms)Flushing 87a15dc17c09c51160706b761e764ba2/cf: appending metadata at 1676214010547 (+45 ms)Flushing 87a15dc17c09c51160706b761e764ba2/cf: closing flushed file at 1676214010547Flushing 87a15dc17c09c51160706b761e764ba2/cf: reopening flushed file at 1676214010579 (+32 ms)Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for 87a15dc17c09c51160706b761e764ba2 in 102ms, sequenceid=8, compaction requested=false at 1676214010588 (+9 ms)Writing region close event to WAL at 1676214010591 (+3 ms)Running coprocessor post-close hooks at 1676214010603 (+12 ms)Closed at 1676214010603 2023-02-12 15:00:10,605 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN in 485 msec 2023-02-12 15:00:10,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:10,606 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSED 2023-02-12 15:00:10,606 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214010606"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214010606"}]},"ts":"1676214010606"} 2023-02-12 15:00:10,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 15:00:10,607 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:10,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e116768b6cb1e8fb57dfa7c68914d39c: Waiting for close lock at 1676214010482Running coprocessor pre-close hooks at 1676214010482Disabling compacts and flushes for region at 1676214010483 (+1 ms)Disabling writes for close at 1676214010483Obtaining lock to block concurrent updates at 1676214010483Preparing flush snapshotting stores in e116768b6cb1e8fb57dfa7c68914d39c at 1676214010483Finished memstore snapshotting testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c., syncing WAL and waiting on mvcc, flushsize=dataSize=22106, getHeapSize=47760, getOffHeapSize=0, getCellsCount=330 at 1676214010484 (+1 ms)Flushing stores of testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. at 1676214010485 (+1 ms)Flushing e116768b6cb1e8fb57dfa7c68914d39c/cf: creating writer at 1676214010485Flushing mob file e116768b6cb1e8fb57dfa7c68914d39c/cf: appending metadata at 1676214010515 (+30 ms)Flushing mob file e116768b6cb1e8fb57dfa7c68914d39c/cf: closing flushed file at 1676214010515Flushing e116768b6cb1e8fb57dfa7c68914d39c/cf: appending metadata at 1676214010555 (+40 ms)Flushing e116768b6cb1e8fb57dfa7c68914d39c/cf: closing flushed file at 1676214010555Flushing e116768b6cb1e8fb57dfa7c68914d39c/cf: reopening flushed file at 1676214010584 (+29 ms)Finished flush of dataSize ~21.59 KB/22106, heapSize ~46.64 KB/47760, currentSize=0 B/0 for e116768b6cb1e8fb57dfa7c68914d39c in 113ms, sequenceid=8, compaction requested=false at 1676214010596 (+12 ms)Writing region close event to WAL at 1676214010602 (+6 ms)Running coprocessor post-close hooks at 1676214010607 (+5 ms)Closed at 1676214010607 2023-02-12 15:00:10,609 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:10,609 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSED 2023-02-12 15:00:10,610 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214010609"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214010609"}]},"ts":"1676214010609"} 2023-02-12 15:00:10,610 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=53 2023-02-12 15:00:10,610 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=53, state=SUCCESS; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,38453,1676214004461 in 484 msec 2023-02-12 15:00:10,612 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN in 493 msec 2023-02-12 15:00:10,614 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=55 2023-02-12 15:00:10,614 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=55, state=SUCCESS; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423 in 484 msec 2023-02-12 15:00:10,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=49 2023-02-12 15:00:10,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN in 497 msec 2023-02-12 15:00:10,617 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214010617"}]},"ts":"1676214010617"} 2023-02-12 15:00:10,619 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLED in hbase:meta 2023-02-12 15:00:10,624 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676214007285 to state=DISABLED 2023-02-12 15:00:10,627 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676214007285 in 519 msec 2023-02-12 15:00:10,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-12 15:00:10,865 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 49 completed 2023-02-12 15:00:10,866 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:10,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676214010866 (current time:1676214010866). 2023-02-12 15:00:10,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 15:00:10,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(139): Connect 0x072657d3 to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:10,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ce3977e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:10,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:10,875 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56814, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:10,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x072657d3 to 127.0.0.1:61893 2023-02-12 15:00:10,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:10,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 15:00:10,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:10,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:10,880 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:10,881 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE; timeout=600000, timestamp=1676214610881 2023-02-12 15:00:10,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:10,881 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676214007285 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676214007285 2023-02-12 15:00:10,882 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 15:00:10,882 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 15:00:10,903 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } 2023-02-12 15:00:10,905 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:10,905 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:10,905 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:10,905 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:10,905 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:10,905 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:10,905 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:10,906 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:10,906 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:10,906 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:10,905 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:10,906 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:10,906 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:10,906 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:10,908 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015] hfiles 2023-02-12 15:00:10,908 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015 2023-02-12 15:00:10,913 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2] hfiles 2023-02-12 15:00:10,913 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2 2023-02-12 15:00:10,913 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4] hfiles 2023-02-12 15:00:10,913 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55] hfiles 2023-02-12 15:00:10,913 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4 2023-02-12 15:00:10,913 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55 2023-02-12 15:00:10,914 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d] hfiles 2023-02-12 15:00:10,914 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d 2023-02-12 15:00:10,916 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6] hfiles 2023-02-12 15:00:10,916 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6 2023-02-12 15:00:10,916 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766] hfiles 2023-02-12 15:00:10,916 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 2023-02-12 15:00:10,916 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 2023-02-12 15:00:10,916 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c 2023-02-12 15:00:10,917 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b 2023-02-12 15:00:10,917 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a 2023-02-12 15:00:10,917 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 2023-02-12 15:00:10,964 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } as finished. 2023-02-12 15:00:10,964 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676214007285 2023-02-12 15:00:10,965 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/snaptb0-1676214007285 2023-02-12 15:00:11,010 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/snaptb0-1676214007285 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/snaptb0-1676214007285 2023-02-12 15:00:11,044 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676214007285 of table testRestoreSnapshot-1676214007285 completed 2023-02-12 15:00:11,044 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/snaptb0-1676214007285 2023-02-12 15:00:11,044 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/snaptb0-1676214007285 2023-02-12 15:00:11,045 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676214007285 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676214007285 at 1676214010882Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } at 1676214010903 (+21 ms)Consolidate snapshot: snaptb0-1676214007285 at 1676214010964 (+61 ms)Loading Region manifests for snaptb0-1676214007285 at 1676214010965 (+1 ms)Writing data manifest for snaptb0-1676214007285 at 1676214010973 (+8 ms)Verifying snapshot: snaptb0-1676214007285 at 1676214010990 (+17 ms)Snapshot snaptb0-1676214007285 of table testRestoreSnapshot-1676214007285 completed at 1676214011044 (+54 ms) 2023-02-12 15:00:11,046 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:11,048 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE in 167 msec 2023-02-12 15:00:11,132 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 15:00:11,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } is done 2023-02-12 15:00:11,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 15:00:11,134 INFO [Listener at localhost/38323] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676214007285 2023-02-12 15:00:11,135 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676214007285 2023-02-12 15:00:11,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:11,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-12 15:00:11,139 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676214007285 2023-02-12 15:00:11,139 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214011139"}]},"ts":"1676214011139"} 2023-02-12 15:00:11,141 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLING in hbase:meta 2023-02-12 15:00:11,148 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 15:00:11,148 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN}] 2023-02-12 15:00:11,150 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN 2023-02-12 15:00:11,150 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN 2023-02-12 15:00:11,150 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN 2023-02-12 15:00:11,150 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN 2023-02-12 15:00:11,150 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN 2023-02-12 15:00:11,151 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:11,151 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=true 2023-02-12 15:00:11,151 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:11,151 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=true 2023-02-12 15:00:11,152 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:11,152 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=true 2023-02-12 15:00:11,152 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN 2023-02-12 15:00:11,152 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:11,152 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:11,152 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=true 2023-02-12 15:00:11,152 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=true 2023-02-12 15:00:11,153 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:11,153 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=true 2023-02-12 15:00:11,301 INFO [jenkins-hbase4:45369] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 15:00:11,302 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:11,302 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:11,302 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214011302"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214011302"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214011302"}]},"ts":"1676214011302"} 2023-02-12 15:00:11,302 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:11,302 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:11,302 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214011302"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214011302"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214011302"}]},"ts":"1676214011302"} 2023-02-12 15:00:11,302 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:11,302 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214011302"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214011302"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214011302"}]},"ts":"1676214011302"} 2023-02-12 15:00:11,302 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214011302"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214011302"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214011302"}]},"ts":"1676214011302"} 2023-02-12 15:00:11,302 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214011302"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214011302"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214011302"}]},"ts":"1676214011302"} 2023-02-12 15:00:11,304 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=68, state=RUNNABLE; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:11,305 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=66, state=RUNNABLE; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:11,306 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=69, state=RUNNABLE; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:11,308 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=67, state=RUNNABLE; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:11,309 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:11,309 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214011309"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214011309"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214011309"}]},"ts":"1676214011309"} 2023-02-12 15:00:11,309 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=65, state=RUNNABLE; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:11,313 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=64, state=RUNNABLE; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:11,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-12 15:00:11,463 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:11,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 15:00:11,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:11,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:11,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:11,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:11,465 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:11,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 15:00:11,466 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:11,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 15:00:11,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:11,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:11,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:11,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:11,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:11,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:11,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:11,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:11,467 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:11,469 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:11,469 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:11,469 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:11,469 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:11,470 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a2b878baabc37b3076d648a696d2d33f columnFamilyName cf 2023-02-12 15:00:11,472 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:11,472 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:11,472 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:11,472 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:11,472 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c70c08e0263941ae1bfbb2d57f1c3052 columnFamilyName cf 2023-02-12 15:00:11,472 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 469de65a1b57113b3e7565d5f5bf8201 columnFamilyName cf 2023-02-12 15:00:11,482 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d 2023-02-12 15:00:11,482 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(310): Store=a2b878baabc37b3076d648a696d2d33f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:11,483 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6 2023-02-12 15:00:11,483 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2 2023-02-12 15:00:11,483 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(310): Store=469de65a1b57113b3e7565d5f5bf8201/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:11,483 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(310): Store=c70c08e0263941ae1bfbb2d57f1c3052/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:11,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:11,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:11,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:11,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:11,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:11,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:11,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:11,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:11,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:11,494 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a2b878baabc37b3076d648a696d2d33f; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68713248, jitterRate=0.023907184600830078}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:11,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a2b878baabc37b3076d648a696d2d33f: Running coprocessor pre-open hook at 1676214011464Writing region info on filesystem at 1676214011464Initializing all the Stores at 1676214011466 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214011466Cleaning up temporary data from old regions at 1676214011486 (+20 ms)Cleaning up detritus from prior splits at 1676214011487 (+1 ms)Running coprocessor post-open hooks at 1676214011494 (+7 ms)Region opened successfully at 1676214011494 2023-02-12 15:00:11,494 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 469de65a1b57113b3e7565d5f5bf8201; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71264869, jitterRate=0.06192930042743683}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:11,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 469de65a1b57113b3e7565d5f5bf8201: Running coprocessor pre-open hook at 1676214011466Writing region info on filesystem at 1676214011466Initializing all the Stores at 1676214011468 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214011468Cleaning up temporary data from old regions at 1676214011488 (+20 ms)Cleaning up detritus from prior splits at 1676214011489 (+1 ms)Running coprocessor post-open hooks at 1676214011494 (+5 ms)Region opened successfully at 1676214011494 2023-02-12 15:00:11,494 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c70c08e0263941ae1bfbb2d57f1c3052; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59300063, jitterRate=-0.11636020243167877}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:11,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c70c08e0263941ae1bfbb2d57f1c3052: Running coprocessor pre-open hook at 1676214011466Writing region info on filesystem at 1676214011466Initializing all the Stores at 1676214011468 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214011468Cleaning up temporary data from old regions at 1676214011486 (+18 ms)Cleaning up detritus from prior splits at 1676214011488 (+2 ms)Running coprocessor post-open hooks at 1676214011495 (+7 ms)Region opened successfully at 1676214011495 2023-02-12 15:00:11,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f., pid=70, masterSystemTime=1676214011456 2023-02-12 15:00:11,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201., pid=75, masterSystemTime=1676214011459 2023-02-12 15:00:11,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052., pid=71, masterSystemTime=1676214011459 2023-02-12 15:00:11,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:11,498 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:11,498 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:11,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 15:00:11,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:11,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:11,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:11,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:11,499 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:11,499 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214011499"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214011499"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214011499"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214011499"}]},"ts":"1676214011499"} 2023-02-12 15:00:11,499 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:11,500 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:11,500 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:11,500 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 15:00:11,500 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:11,500 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:11,500 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:11,500 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:11,501 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:11,501 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:11,501 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214011501"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214011501"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214011501"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214011501"}]},"ts":"1676214011501"} 2023-02-12 15:00:11,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:11,502 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:11,502 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:11,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 15:00:11,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:11,502 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:11,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:11,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:11,503 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214011502"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214011502"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214011502"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214011502"}]},"ts":"1676214011502"} 2023-02-12 15:00:11,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:11,503 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:11,503 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:11,503 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:11,504 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e31ecbd7855a2e0bca06cbe75ecfc8fe columnFamilyName cf 2023-02-12 15:00:11,505 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:11,505 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:11,505 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:11,506 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e116768b6cb1e8fb57dfa7c68914d39c columnFamilyName cf 2023-02-12 15:00:11,507 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=68 2023-02-12 15:00:11,507 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=68, state=SUCCESS; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,37113,1676214004377 in 198 msec 2023-02-12 15:00:11,508 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:11,508 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:11,508 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=64 2023-02-12 15:00:11,508 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 87a15dc17c09c51160706b761e764ba2 columnFamilyName cf 2023-02-12 15:00:11,508 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=64, state=SUCCESS; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,33845,1676214004423 in 193 msec 2023-02-12 15:00:11,509 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=66 2023-02-12 15:00:11,509 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN in 359 msec 2023-02-12 15:00:11,509 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=66, state=SUCCESS; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461 in 201 msec 2023-02-12 15:00:11,510 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN in 360 msec 2023-02-12 15:00:11,511 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN in 361 msec 2023-02-12 15:00:11,518 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55 2023-02-12 15:00:11,518 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4 2023-02-12 15:00:11,518 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(310): Store=87a15dc17c09c51160706b761e764ba2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:11,518 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(310): Store=e116768b6cb1e8fb57dfa7c68914d39c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:11,518 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015 2023-02-12 15:00:11,518 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(310): Store=e31ecbd7855a2e0bca06cbe75ecfc8fe/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:11,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:11,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:11,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:11,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:11,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:11,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:11,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:11,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:11,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:11,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 87a15dc17c09c51160706b761e764ba2; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72477492, jitterRate=0.07999879121780396}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:11,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 87a15dc17c09c51160706b761e764ba2: Running coprocessor pre-open hook at 1676214011503Writing region info on filesystem at 1676214011503Initializing all the Stores at 1676214011505 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214011505Cleaning up temporary data from old regions at 1676214011521 (+16 ms)Cleaning up detritus from prior splits at 1676214011522 (+1 ms)Running coprocessor post-open hooks at 1676214011526 (+4 ms)Region opened successfully at 1676214011526 2023-02-12 15:00:11,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e116768b6cb1e8fb57dfa7c68914d39c; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60590676, jitterRate=-0.09712857007980347}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:11,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e116768b6cb1e8fb57dfa7c68914d39c: Running coprocessor pre-open hook at 1676214011500Writing region info on filesystem at 1676214011500Initializing all the Stores at 1676214011503 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214011503Cleaning up temporary data from old regions at 1676214011522 (+19 ms)Cleaning up detritus from prior splits at 1676214011522Running coprocessor post-open hooks at 1676214011526 (+4 ms)Region opened successfully at 1676214011526 2023-02-12 15:00:11,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e31ecbd7855a2e0bca06cbe75ecfc8fe; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65295338, jitterRate=-0.027023643255233765}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:11,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Running coprocessor pre-open hook at 1676214011498Writing region info on filesystem at 1676214011498Initializing all the Stores at 1676214011500 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214011500Cleaning up temporary data from old regions at 1676214011521 (+21 ms)Cleaning up detritus from prior splits at 1676214011522 (+1 ms)Running coprocessor post-open hooks at 1676214011527 (+5 ms)Region opened successfully at 1676214011527 2023-02-12 15:00:11,527 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe., pid=74, masterSystemTime=1676214011456 2023-02-12 15:00:11,527 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2., pid=73, masterSystemTime=1676214011459 2023-02-12 15:00:11,527 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c., pid=72, masterSystemTime=1676214011459 2023-02-12 15:00:11,530 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:11,530 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:11,530 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:11,531 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214011530"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214011530"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214011530"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214011530"}]},"ts":"1676214011530"} 2023-02-12 15:00:11,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:11,531 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:11,531 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:11,532 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214011531"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214011531"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214011531"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214011531"}]},"ts":"1676214011531"} 2023-02-12 15:00:11,532 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:11,532 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:11,533 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:11,533 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214011533"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214011533"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214011533"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214011533"}]},"ts":"1676214011533"} 2023-02-12 15:00:11,536 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=65 2023-02-12 15:00:11,536 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=65, state=SUCCESS; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377 in 224 msec 2023-02-12 15:00:11,537 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=69 2023-02-12 15:00:11,537 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=69, state=SUCCESS; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423 in 229 msec 2023-02-12 15:00:11,539 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN in 388 msec 2023-02-12 15:00:11,539 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=67 2023-02-12 15:00:11,539 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=67, state=SUCCESS; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,38453,1676214004461 in 228 msec 2023-02-12 15:00:11,539 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN in 389 msec 2023-02-12 15:00:11,541 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=67, resume processing ppid=63 2023-02-12 15:00:11,541 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN in 391 msec 2023-02-12 15:00:11,542 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214011542"}]},"ts":"1676214011542"} 2023-02-12 15:00:11,543 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLED in hbase:meta 2023-02-12 15:00:11,545 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676214007285' was successfully enabled. 2023-02-12 15:00:11,548 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676214007285 in 411 msec 2023-02-12 15:00:11,749 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-12 15:00:11,802 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-12 15:00:11,803 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-12 15:00:11,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-12 15:00:11,891 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 63 completed 2023-02-12 15:00:11,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:11,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:11,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:11,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:11,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:11,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:11,914 DEBUG [Listener at localhost/38323] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676214007285 2023-02-12 15:00:11,914 DEBUG [Listener at localhost/38323] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:11,916 INFO [Listener at localhost/38323] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-12 15:00:12,182 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 15:00:12,373 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 15:00:12,511 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 15:00:12,668 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 15:00:12,806 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 15:00:13,350 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-12 15:00:13,351 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-12 15:00:14,160 DEBUG [Listener at localhost/38323] zookeeper.ReadOnlyZKClient(139): Connect 0x700455dd to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:14,166 DEBUG [Listener at localhost/38323] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6d06cc97, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:14,168 DEBUG [hconnection-0x490facbf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:14,171 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43802, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:14,175 DEBUG [hconnection-0x490facbf-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:14,179 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55980, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:14,317 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 15:00:14,462 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 15:00:14,466 DEBUG [hconnection-0x490facbf-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:14,470 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59100, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:14,572 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 15:00:14,693 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 15:00:14,810 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 15:00:15,982 INFO [Listener at localhost/38323] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676214007285 2023-02-12 15:00:15,983 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676214007285 2023-02-12 15:00:15,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:15,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-12 15:00:15,987 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214015987"}]},"ts":"1676214015987"} 2023-02-12 15:00:15,989 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLING in hbase:meta 2023-02-12 15:00:15,991 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676214007285 to state=DISABLING 2023-02-12 15:00:15,992 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN}] 2023-02-12 15:00:15,994 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN 2023-02-12 15:00:15,994 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN 2023-02-12 15:00:15,994 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN 2023-02-12 15:00:15,994 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN 2023-02-12 15:00:15,994 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:15,995 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN 2023-02-12 15:00:15,995 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:15,995 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214015994"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214015994"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214015994"}]},"ts":"1676214015994"} 2023-02-12 15:00:15,995 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214015995"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214015995"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214015995"}]},"ts":"1676214015995"} 2023-02-12 15:00:15,995 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:15,995 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:15,995 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:15,996 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214015995"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214015995"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214015995"}]},"ts":"1676214015995"} 2023-02-12 15:00:15,996 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214015995"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214015995"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214015995"}]},"ts":"1676214015995"} 2023-02-12 15:00:15,996 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214015995"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214015995"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214015995"}]},"ts":"1676214015995"} 2023-02-12 15:00:15,997 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=78, state=RUNNABLE; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:15,997 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=77, state=RUNNABLE; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:15,998 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=81, state=RUNNABLE; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:15,999 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=80, state=RUNNABLE; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:15,999 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=82, state=RUNNABLE; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:16,002 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN 2023-02-12 15:00:16,003 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:16,003 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214016003"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214016003"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214016003"}]},"ts":"1676214016003"} 2023-02-12 15:00:16,005 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=79, state=RUNNABLE; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:16,151 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:16,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e31ecbd7855a2e0bca06cbe75ecfc8fe, disabling compactions & flushes 2023-02-12 15:00:16,152 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:16,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:16,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. after waiting 0 ms 2023-02-12 15:00:16,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:16,152 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:16,152 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:16,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 87a15dc17c09c51160706b761e764ba2, disabling compactions & flushes 2023-02-12 15:00:16,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 469de65a1b57113b3e7565d5f5bf8201, disabling compactions & flushes 2023-02-12 15:00:16,152 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e31ecbd7855a2e0bca06cbe75ecfc8fe 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-12 15:00:16,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:16,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:16,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:16,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:16,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. after waiting 0 ms 2023-02-12 15:00:16,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. after waiting 0 ms 2023-02-12 15:00:16,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:16,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:16,154 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 87a15dc17c09c51160706b761e764ba2 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-12 15:00:16,154 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 469de65a1b57113b3e7565d5f5bf8201 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-12 15:00:16,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 2023-02-12 15:00:16,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 2023-02-12 15:00:16,188 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f 2023-02-12 15:00:16,190 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/.tmp/cf/984b06764f54429d98a03a4b3659e10a 2023-02-12 15:00:16,191 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/.tmp/cf/a67854a9941340869b744861955b1505 2023-02-12 15:00:16,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/.tmp/cf/984b06764f54429d98a03a4b3659e10a as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/984b06764f54429d98a03a4b3659e10a 2023-02-12 15:00:16,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/.tmp/cf/a67854a9941340869b744861955b1505 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/a67854a9941340869b744861955b1505 2023-02-12 15:00:16,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/984b06764f54429d98a03a4b3659e10a, entries=29, sequenceid=15, filesize=9.8 K 2023-02-12 15:00:16,200 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 469de65a1b57113b3e7565d5f5bf8201 in 45ms, sequenceid=15, compaction requested=false 2023-02-12 15:00:16,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/a67854a9941340869b744861955b1505, entries=31, sequenceid=15, filesize=10.2 K 2023-02-12 15:00:16,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for e31ecbd7855a2e0bca06cbe75ecfc8fe in 49ms, sequenceid=15, compaction requested=false 2023-02-12 15:00:16,215 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 15:00:16,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:16,216 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 469de65a1b57113b3e7565d5f5bf8201: Waiting for close lock at 1676214016153Running coprocessor pre-close hooks at 1676214016153Disabling compacts and flushes for region at 1676214016153Disabling writes for close at 1676214016154 (+1 ms)Obtaining lock to block concurrent updates at 1676214016154Preparing flush snapshotting stores in 469de65a1b57113b3e7565d5f5bf8201 at 1676214016154Finished memstore snapshotting testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201., syncing WAL and waiting on mvcc, flushsize=dataSize=1941, getHeapSize=4416, getOffHeapSize=0, getCellsCount=29 at 1676214016154Flushing stores of testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. at 1676214016155 (+1 ms)Flushing 469de65a1b57113b3e7565d5f5bf8201/cf: creating writer at 1676214016156 (+1 ms)Flushing mob file 469de65a1b57113b3e7565d5f5bf8201/cf: appending metadata at 1676214016165 (+9 ms)Flushing mob file 469de65a1b57113b3e7565d5f5bf8201/cf: closing flushed file at 1676214016165Flushing 469de65a1b57113b3e7565d5f5bf8201/cf: appending metadata at 1676214016183 (+18 ms)Flushing 469de65a1b57113b3e7565d5f5bf8201/cf: closing flushed file at 1676214016183Flushing 469de65a1b57113b3e7565d5f5bf8201/cf: reopening flushed file at 1676214016195 (+12 ms)Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 469de65a1b57113b3e7565d5f5bf8201 in 45ms, sequenceid=15, compaction requested=false at 1676214016200 (+5 ms)Writing region close event to WAL at 1676214016211 (+11 ms)Running coprocessor post-close hooks at 1676214016215 (+4 ms)Closed at 1676214016216 (+1 ms) 2023-02-12 15:00:16,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/.tmp/cf/393173ee6d644dfbb763f3ba123cdb44 2023-02-12 15:00:16,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:16,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 15:00:16,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:16,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e116768b6cb1e8fb57dfa7c68914d39c, disabling compactions & flushes 2023-02-12 15:00:16,219 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSED 2023-02-12 15:00:16,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:16,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:16,219 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214016219"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214016219"}]},"ts":"1676214016219"} 2023-02-12 15:00:16,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. after waiting 0 ms 2023-02-12 15:00:16,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:16,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e116768b6cb1e8fb57dfa7c68914d39c 1/1 column families, dataSize=22.96 KB heapSize=49.61 KB 2023-02-12 15:00:16,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:16,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Waiting for close lock at 1676214016151Running coprocessor pre-close hooks at 1676214016151Disabling compacts and flushes for region at 1676214016151Disabling writes for close at 1676214016152 (+1 ms)Obtaining lock to block concurrent updates at 1676214016153 (+1 ms)Preparing flush snapshotting stores in e31ecbd7855a2e0bca06cbe75ecfc8fe at 1676214016153Finished memstore snapshotting testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676214016154 (+1 ms)Flushing stores of testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. at 1676214016154Flushing e31ecbd7855a2e0bca06cbe75ecfc8fe/cf: creating writer at 1676214016155 (+1 ms)Flushing mob file e31ecbd7855a2e0bca06cbe75ecfc8fe/cf: appending metadata at 1676214016171 (+16 ms)Flushing mob file e31ecbd7855a2e0bca06cbe75ecfc8fe/cf: closing flushed file at 1676214016171Flushing e31ecbd7855a2e0bca06cbe75ecfc8fe/cf: appending metadata at 1676214016183 (+12 ms)Flushing e31ecbd7855a2e0bca06cbe75ecfc8fe/cf: closing flushed file at 1676214016183Flushing e31ecbd7855a2e0bca06cbe75ecfc8fe/cf: reopening flushed file at 1676214016197 (+14 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for e31ecbd7855a2e0bca06cbe75ecfc8fe in 49ms, sequenceid=15, compaction requested=false at 1676214016201 (+4 ms)Writing region close event to WAL at 1676214016211 (+10 ms)Running coprocessor post-close hooks at 1676214016220 (+9 ms)Closed at 1676214016220 2023-02-12 15:00:16,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:16,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:16,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a2b878baabc37b3076d648a696d2d33f, disabling compactions & flushes 2023-02-12 15:00:16,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:16,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:16,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. after waiting 0 ms 2023-02-12 15:00:16,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:16,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing a2b878baabc37b3076d648a696d2d33f 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-12 15:00:16,225 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSED 2023-02-12 15:00:16,226 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214016225"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214016225"}]},"ts":"1676214016225"} 2023-02-12 15:00:16,227 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/.tmp/cf/393173ee6d644dfbb763f3ba123cdb44 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/393173ee6d644dfbb763f3ba123cdb44 2023-02-12 15:00:16,227 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=77 2023-02-12 15:00:16,227 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=77, state=SUCCESS; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,33845,1676214004423 in 225 msec 2023-02-12 15:00:16,229 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN in 235 msec 2023-02-12 15:00:16,230 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=78 2023-02-12 15:00:16,230 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=78, state=SUCCESS; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377 in 231 msec 2023-02-12 15:00:16,232 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN in 238 msec 2023-02-12 15:00:16,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/393173ee6d644dfbb763f3ba123cdb44, entries=28, sequenceid=15, filesize=9.7 K 2023-02-12 15:00:16,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 87a15dc17c09c51160706b761e764ba2 in 85ms, sequenceid=15, compaction requested=false 2023-02-12 15:00:16,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-12 15:00:16,261 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 15:00:16,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:16,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 87a15dc17c09c51160706b761e764ba2: Waiting for close lock at 1676214016152Running coprocessor pre-close hooks at 1676214016152Disabling compacts and flushes for region at 1676214016152Disabling writes for close at 1676214016154 (+2 ms)Obtaining lock to block concurrent updates at 1676214016154Preparing flush snapshotting stores in 87a15dc17c09c51160706b761e764ba2 at 1676214016154Finished memstore snapshotting testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2., syncing WAL and waiting on mvcc, flushsize=dataSize=1874, getHeapSize=4272, getOffHeapSize=0, getCellsCount=28 at 1676214016154Flushing stores of testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. at 1676214016155 (+1 ms)Flushing 87a15dc17c09c51160706b761e764ba2/cf: creating writer at 1676214016156 (+1 ms)Flushing mob file 87a15dc17c09c51160706b761e764ba2/cf: appending metadata at 1676214016171 (+15 ms)Flushing mob file 87a15dc17c09c51160706b761e764ba2/cf: closing flushed file at 1676214016171Flushing 87a15dc17c09c51160706b761e764ba2/cf: appending metadata at 1676214016189 (+18 ms)Flushing 87a15dc17c09c51160706b761e764ba2/cf: closing flushed file at 1676214016189Flushing 87a15dc17c09c51160706b761e764ba2/cf: reopening flushed file at 1676214016228 (+39 ms)Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 87a15dc17c09c51160706b761e764ba2 in 85ms, sequenceid=15, compaction requested=false at 1676214016239 (+11 ms)Writing region close event to WAL at 1676214016247 (+8 ms)Running coprocessor post-close hooks at 1676214016262 (+15 ms)Closed at 1676214016263 (+1 ms) 2023-02-12 15:00:16,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:16,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:16,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c70c08e0263941ae1bfbb2d57f1c3052, disabling compactions & flushes 2023-02-12 15:00:16,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:16,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:16,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. after waiting 0 ms 2023-02-12 15:00:16,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:16,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c70c08e0263941ae1bfbb2d57f1c3052 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-12 15:00:16,268 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSED 2023-02-12 15:00:16,268 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 2023-02-12 15:00:16,268 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214016268"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214016268"}]},"ts":"1676214016268"} 2023-02-12 15:00:16,272 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=80 2023-02-12 15:00:16,272 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=80, state=SUCCESS; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,38453,1676214004461 in 271 msec 2023-02-12 15:00:16,275 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN in 280 msec 2023-02-12 15:00:16,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=23.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/.tmp/cf/49b9c7e6778343f982623f169324b842 2023-02-12 15:00:16,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/.tmp/cf/49b9c7e6778343f982623f169324b842 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/49b9c7e6778343f982623f169324b842 2023-02-12 15:00:16,294 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 2023-02-12 15:00:16,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/49b9c7e6778343f982623f169324b842, entries=351, sequenceid=15, filesize=64.7 K 2023-02-12 15:00:16,299 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.96 KB/23513, heapSize ~49.59 KB/50784, currentSize=0 B/0 for e116768b6cb1e8fb57dfa7c68914d39c in 80ms, sequenceid=15, compaction requested=false 2023-02-12 15:00:16,315 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/.tmp/cf/3a1b76d129894ec2adcd309280852eb4 2023-02-12 15:00:16,316 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 15:00:16,317 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:16,317 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e116768b6cb1e8fb57dfa7c68914d39c: Waiting for close lock at 1676214016219Running coprocessor pre-close hooks at 1676214016219Disabling compacts and flushes for region at 1676214016219Disabling writes for close at 1676214016219Obtaining lock to block concurrent updates at 1676214016219Preparing flush snapshotting stores in e116768b6cb1e8fb57dfa7c68914d39c at 1676214016219Finished memstore snapshotting testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c., syncing WAL and waiting on mvcc, flushsize=dataSize=23513, getHeapSize=50784, getOffHeapSize=0, getCellsCount=351 at 1676214016220 (+1 ms)Flushing stores of testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. at 1676214016221 (+1 ms)Flushing e116768b6cb1e8fb57dfa7c68914d39c/cf: creating writer at 1676214016221Flushing mob file e116768b6cb1e8fb57dfa7c68914d39c/cf: appending metadata at 1676214016233 (+12 ms)Flushing mob file e116768b6cb1e8fb57dfa7c68914d39c/cf: closing flushed file at 1676214016233Flushing e116768b6cb1e8fb57dfa7c68914d39c/cf: appending metadata at 1676214016273 (+40 ms)Flushing e116768b6cb1e8fb57dfa7c68914d39c/cf: closing flushed file at 1676214016273Flushing e116768b6cb1e8fb57dfa7c68914d39c/cf: reopening flushed file at 1676214016293 (+20 ms)Finished flush of dataSize ~22.96 KB/23513, heapSize ~49.59 KB/50784, currentSize=0 B/0 for e116768b6cb1e8fb57dfa7c68914d39c in 80ms, sequenceid=15, compaction requested=false at 1676214016299 (+6 ms)Writing region close event to WAL at 1676214016311 (+12 ms)Running coprocessor post-close hooks at 1676214016316 (+5 ms)Closed at 1676214016317 (+1 ms) 2023-02-12 15:00:16,319 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:16,320 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSED 2023-02-12 15:00:16,320 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214016320"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214016320"}]},"ts":"1676214016320"} 2023-02-12 15:00:16,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/.tmp/cf/3a1b76d129894ec2adcd309280852eb4 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/3a1b76d129894ec2adcd309280852eb4 2023-02-12 15:00:16,351 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x700455dd to 127.0.0.1:61893 2023-02-12 15:00:16,351 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:16,353 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=82 2023-02-12 15:00:16,354 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=82, state=SUCCESS; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423 in 350 msec 2023-02-12 15:00:16,356 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN in 362 msec 2023-02-12 15:00:16,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/3a1b76d129894ec2adcd309280852eb4, entries=27, sequenceid=15, filesize=9.5 K 2023-02-12 15:00:16,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for c70c08e0263941ae1bfbb2d57f1c3052 in 91ms, sequenceid=15, compaction requested=false 2023-02-12 15:00:16,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 15:00:16,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:16,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c70c08e0263941ae1bfbb2d57f1c3052: Waiting for close lock at 1676214016265Running coprocessor pre-close hooks at 1676214016265Disabling compacts and flushes for region at 1676214016265Disabling writes for close at 1676214016266 (+1 ms)Obtaining lock to block concurrent updates at 1676214016266Preparing flush snapshotting stores in c70c08e0263941ae1bfbb2d57f1c3052 at 1676214016266Finished memstore snapshotting testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676214016267 (+1 ms)Flushing stores of testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. at 1676214016268 (+1 ms)Flushing c70c08e0263941ae1bfbb2d57f1c3052/cf: creating writer at 1676214016269 (+1 ms)Flushing mob file c70c08e0263941ae1bfbb2d57f1c3052/cf: appending metadata at 1676214016276 (+7 ms)Flushing mob file c70c08e0263941ae1bfbb2d57f1c3052/cf: closing flushed file at 1676214016276Flushing c70c08e0263941ae1bfbb2d57f1c3052/cf: appending metadata at 1676214016295 (+19 ms)Flushing c70c08e0263941ae1bfbb2d57f1c3052/cf: closing flushed file at 1676214016295Flushing c70c08e0263941ae1bfbb2d57f1c3052/cf: reopening flushed file at 1676214016350 (+55 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for c70c08e0263941ae1bfbb2d57f1c3052 in 91ms, sequenceid=15, compaction requested=false at 1676214016357 (+7 ms)Writing region close event to WAL at 1676214016360 (+3 ms)Running coprocessor post-close hooks at 1676214016364 (+4 ms)Closed at 1676214016365 (+1 ms) 2023-02-12 15:00:16,366 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:16,367 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSED 2023-02-12 15:00:16,367 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214016367"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214016367"}]},"ts":"1676214016367"} 2023-02-12 15:00:16,371 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=79 2023-02-12 15:00:16,371 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=79, state=SUCCESS; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461 in 364 msec 2023-02-12 15:00:16,372 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN in 379 msec 2023-02-12 15:00:16,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 2023-02-12 15:00:16,674 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/.tmp/cf/9837fd0f2c07404bbfa672c073f42881 2023-02-12 15:00:16,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/.tmp/cf/9837fd0f2c07404bbfa672c073f42881 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/9837fd0f2c07404bbfa672c073f42881 2023-02-12 15:00:16,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/9837fd0f2c07404bbfa672c073f42881, entries=34, sequenceid=15, filesize=10.7 K 2023-02-12 15:00:16,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for a2b878baabc37b3076d648a696d2d33f in 459ms, sequenceid=15, compaction requested=false 2023-02-12 15:00:16,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 15:00:16,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:16,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a2b878baabc37b3076d648a696d2d33f: Waiting for close lock at 1676214016223Running coprocessor pre-close hooks at 1676214016223Disabling compacts and flushes for region at 1676214016223Disabling writes for close at 1676214016224 (+1 ms)Obtaining lock to block concurrent updates at 1676214016224Preparing flush snapshotting stores in a2b878baabc37b3076d648a696d2d33f at 1676214016224Finished memstore snapshotting testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f., syncing WAL and waiting on mvcc, flushsize=dataSize=2276, getHeapSize=5136, getOffHeapSize=0, getCellsCount=34 at 1676214016224Flushing stores of testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. at 1676214016225 (+1 ms)Flushing a2b878baabc37b3076d648a696d2d33f/cf: creating writer at 1676214016226 (+1 ms)Flushing mob file a2b878baabc37b3076d648a696d2d33f/cf: appending metadata at 1676214016236 (+10 ms)Flushing mob file a2b878baabc37b3076d648a696d2d33f/cf: closing flushed file at 1676214016236Flushing a2b878baabc37b3076d648a696d2d33f/cf: appending metadata at 1676214016666 (+430 ms)Flushing a2b878baabc37b3076d648a696d2d33f/cf: closing flushed file at 1676214016666Flushing a2b878baabc37b3076d648a696d2d33f/cf: reopening flushed file at 1676214016678 (+12 ms)Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for a2b878baabc37b3076d648a696d2d33f in 459ms, sequenceid=15, compaction requested=false at 1676214016683 (+5 ms)Writing region close event to WAL at 1676214016687 (+4 ms)Running coprocessor post-close hooks at 1676214016691 (+4 ms)Closed at 1676214016691 2023-02-12 15:00:16,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:16,694 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSED 2023-02-12 15:00:16,694 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214016694"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214016694"}]},"ts":"1676214016694"} 2023-02-12 15:00:16,697 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=81 2023-02-12 15:00:16,697 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=81, state=SUCCESS; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,37113,1676214004377 in 698 msec 2023-02-12 15:00:16,699 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=81, resume processing ppid=76 2023-02-12 15:00:16,699 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN in 705 msec 2023-02-12 15:00:16,700 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214016700"}]},"ts":"1676214016700"} 2023-02-12 15:00:16,701 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLED in hbase:meta 2023-02-12 15:00:16,704 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676214007285 to state=DISABLED 2023-02-12 15:00:16,707 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676214007285 in 722 msec 2023-02-12 15:00:16,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-12 15:00:16,741 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 76 completed 2023-02-12 15:00:16,742 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb1-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:16,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676214016742 (current time:1676214016742). 2023-02-12 15:00:16,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 15:00:16,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(139): Connect 0x1ae89f39 to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:16,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2fc5db78, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:16,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:16,752 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43804, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:16,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ae89f39 to 127.0.0.1:61893 2023-02-12 15:00:16,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:16,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 15:00:16,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:16,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:16,756 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:16,757 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE; timeout=600000, timestamp=1676214616757 2023-02-12 15:00:16,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:16,757 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676214007285 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676214007285 2023-02-12 15:00:16,757 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 15:00:16,758 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 15:00:16,772 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } 2023-02-12 15:00:16,777 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:16,777 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:16,777 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:16,777 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:16,777 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:16,778 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:16,778 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:16,778 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:16,778 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:16,778 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:16,778 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:16,778 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:16,778 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:16,778 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:16,780 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/a67854a9941340869b744861955b1505] hfiles 2023-02-12 15:00:16,780 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015 2023-02-12 15:00:16,781 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/984b06764f54429d98a03a4b3659e10a, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6] hfiles 2023-02-12 15:00:16,781 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/a67854a9941340869b744861955b1505 2023-02-12 15:00:16,781 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/984b06764f54429d98a03a4b3659e10a 2023-02-12 15:00:16,781 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6 2023-02-12 15:00:16,781 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/49b9c7e6778343f982623f169324b842, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4] hfiles 2023-02-12 15:00:16,781 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/49b9c7e6778343f982623f169324b842 2023-02-12 15:00:16,781 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4 2023-02-12 15:00:16,782 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/393173ee6d644dfbb763f3ba123cdb44, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55] hfiles 2023-02-12 15:00:16,783 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/393173ee6d644dfbb763f3ba123cdb44 2023-02-12 15:00:16,783 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55 2023-02-12 15:00:16,783 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/3a1b76d129894ec2adcd309280852eb4, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2] hfiles 2023-02-12 15:00:16,783 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/3a1b76d129894ec2adcd309280852eb4 2023-02-12 15:00:16,783 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2 2023-02-12 15:00:16,785 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/9837fd0f2c07404bbfa672c073f42881, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d] hfiles 2023-02-12 15:00:16,785 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/9837fd0f2c07404bbfa672c073f42881 2023-02-12 15:00:16,785 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d 2023-02-12 15:00:16,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f] hfiles 2023-02-12 15:00:16,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 2023-02-12 15:00:16,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 2023-02-12 15:00:16,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 2023-02-12 15:00:16,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 2023-02-12 15:00:16,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c 2023-02-12 15:00:16,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 2023-02-12 15:00:16,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 2023-02-12 15:00:16,789 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b 2023-02-12 15:00:16,790 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a 2023-02-12 15:00:16,790 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 2023-02-12 15:00:16,790 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 2023-02-12 15:00:16,790 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f 2023-02-12 15:00:16,966 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-12 15:00:17,008 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 15:00:17,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } is done 2023-02-12 15:00:17,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(404): Snapshoting '{ ss=snaptb1-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }' is still in progress! 2023-02-12 15:00:17,009 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-12 15:00:17,239 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } as finished. 2023-02-12 15:00:17,240 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676214007285 2023-02-12 15:00:17,240 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/snaptb1-1676214007285 2023-02-12 15:00:17,269 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/snaptb1-1676214007285 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/snaptb1-1676214007285 2023-02-12 15:00:17,289 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676214007285 of table testRestoreSnapshot-1676214007285 completed 2023-02-12 15:00:17,289 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/snaptb1-1676214007285 2023-02-12 15:00:17,289 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/snaptb1-1676214007285 2023-02-12 15:00:17,289 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676214007285 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676214007285 at 1676214016757Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } at 1676214016772 (+15 ms)Consolidate snapshot: snaptb1-1676214007285 at 1676214017240 (+468 ms)Loading Region manifests for snaptb1-1676214007285 at 1676214017240Writing data manifest for snaptb1-1676214007285 at 1676214017245 (+5 ms)Verifying snapshot: snaptb1-1676214007285 at 1676214017254 (+9 ms)Snapshot snaptb1-1676214007285 of table testRestoreSnapshot-1676214007285 completed at 1676214017289 (+35 ms) 2023-02-12 15:00:17,291 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:17,292 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE in 536 msec 2023-02-12 15:00:17,509 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 15:00:17,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } is done 2023-02-12 15:00:17,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676214007285 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 15:00:17,527 INFO [Listener at localhost/38323] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676214007285-1676214017527 2023-02-12 15:00:17,528 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676214007285-1676214017527 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:17,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676214017528 (current time:1676214017528). 2023-02-12 15:00:17,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 15:00:17,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(139): Connect 0x0c6bb90d to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:17,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@637d4a58, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:17,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:17,536 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43812, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:17,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c6bb90d to 127.0.0.1:61893 2023-02-12 15:00:17,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:17,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 15:00:17,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676214007285-1676214017527 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:17,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:17,540 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:17,540 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE; timeout=600000, timestamp=1676214617540 2023-02-12 15:00:17,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676214007285-1676214017527 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:17,540 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676214007285-1676214017527 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676214007285 2023-02-12 15:00:17,541 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676214007285-1676214017527 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 15:00:17,541 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 15:00:17,551 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676214007285-1676214017527 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } 2023-02-12 15:00:17,555 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:17,555 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:17,555 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:17,555 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:17,556 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:17,556 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:17,555 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:17,555 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:17,556 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:17,556 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:17,556 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:17,556 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:17,556 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:17,556 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:17,559 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/a67854a9941340869b744861955b1505] hfiles 2023-02-12 15:00:17,559 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015 2023-02-12 15:00:17,559 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/a67854a9941340869b744861955b1505 2023-02-12 15:00:17,560 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/9837fd0f2c07404bbfa672c073f42881, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d] hfiles 2023-02-12 15:00:17,560 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/9837fd0f2c07404bbfa672c073f42881 2023-02-12 15:00:17,560 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d 2023-02-12 15:00:17,560 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/3a1b76d129894ec2adcd309280852eb4, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2] hfiles 2023-02-12 15:00:17,561 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/3a1b76d129894ec2adcd309280852eb4 2023-02-12 15:00:17,561 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2 2023-02-12 15:00:17,561 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/393173ee6d644dfbb763f3ba123cdb44, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55] hfiles 2023-02-12 15:00:17,561 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/393173ee6d644dfbb763f3ba123cdb44 2023-02-12 15:00:17,561 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55 2023-02-12 15:00:17,561 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/49b9c7e6778343f982623f169324b842, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4] hfiles 2023-02-12 15:00:17,561 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/984b06764f54429d98a03a4b3659e10a, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6] hfiles 2023-02-12 15:00:17,561 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/49b9c7e6778343f982623f169324b842 2023-02-12 15:00:17,561 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/984b06764f54429d98a03a4b3659e10a 2023-02-12 15:00:17,561 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4 2023-02-12 15:00:17,561 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6 2023-02-12 15:00:17,569 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f] hfiles 2023-02-12 15:00:17,569 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 2023-02-12 15:00:17,569 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 2023-02-12 15:00:17,569 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 2023-02-12 15:00:17,570 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 2023-02-12 15:00:17,570 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c 2023-02-12 15:00:17,570 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 2023-02-12 15:00:17,570 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 2023-02-12 15:00:17,570 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b 2023-02-12 15:00:17,570 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a 2023-02-12 15:00:17,570 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 2023-02-12 15:00:17,570 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 2023-02-12 15:00:17,570 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f 2023-02-12 15:00:17,594 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676214007285-1676214017527 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } as finished. 2023-02-12 15:00:17,594 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676214007285-1676214017527 2023-02-12 15:00:17,594 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676214007285-1676214017527 2023-02-12 15:00:17,633 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676214007285-1676214017527 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/hbase-failsafe-snaptb0-1676214007285-1676214017527 2023-02-12 15:00:17,655 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676214007285-1676214017527 of table testRestoreSnapshot-1676214007285 completed 2023-02-12 15:00:17,655 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676214007285-1676214017527 2023-02-12 15:00:17,656 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676214007285-1676214017527 2023-02-12 15:00:17,656 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676214007285-1676214017527 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676214007285 at 1676214017541Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676214007285-1676214017527 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } at 1676214017551 (+10 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676214007285-1676214017527 at 1676214017594 (+43 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676214007285-1676214017527 at 1676214017594Writing data manifest for hbase-failsafe-snaptb0-1676214007285-1676214017527 at 1676214017607 (+13 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676214007285-1676214017527 at 1676214017616 (+9 ms)Snapshot hbase-failsafe-snaptb0-1676214007285-1676214017527 of table testRestoreSnapshot-1676214007285 completed at 1676214017655 (+39 ms) 2023-02-12 15:00:17,657 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:17,659 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE in 119 msec 2023-02-12 15:00:17,791 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 15:00:17,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676214007285-1676214017527 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } is done 2023-02-12 15:00:17,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676214007285-1676214017527 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 15:00:17,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676214007285 snapshot=name: "snaptb0-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214010866 type: DISABLED version: 2 ttl: 0 ) 2023-02-12 15:00:17,820 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676214007285 as table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:17,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-12 15:00:17,838 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/.tabledesc/.tableinfo.0000000002 2023-02-12 15:00:17,839 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/.tabledesc/.tableinfo.0000000001 2023-02-12 15:00:17,839 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/.tabledesc/.tableinfo.0000000002 2023-02-12 15:00:17,840 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } 2023-02-12 15:00:17,845 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214010866 type: DISABLED version: 2 ttl: 0 2023-02-12 15:00:17,846 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285 2023-02-12 15:00:17,854 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:17,862 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 2023-02-12 15:00:17,863 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 2023-02-12 15:00:17,865 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 2023-02-12 15:00:17,866 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 2023-02-12 15:00:17,868 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 2023-02-12 15:00:17,869 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f 2023-02-12 15:00:17,869 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:17,869 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:17,869 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:17,869 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:17,869 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:17,869 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:17,874 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/393173ee6d644dfbb763f3ba123cdb44 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/393173ee6d644dfbb763f3ba123cdb44 2023-02-12 15:00:17,875 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/3a1b76d129894ec2adcd309280852eb4 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/3a1b76d129894ec2adcd309280852eb4 2023-02-12 15:00:17,875 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/9837fd0f2c07404bbfa672c073f42881 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/9837fd0f2c07404bbfa672c073f42881 2023-02-12 15:00:17,876 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/984b06764f54429d98a03a4b3659e10a to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/984b06764f54429d98a03a4b3659e10a 2023-02-12 15:00:17,876 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/a67854a9941340869b744861955b1505 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/a67854a9941340869b744861955b1505 2023-02-12 15:00:17,876 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/49b9c7e6778343f982623f169324b842 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/49b9c7e6778343f982623f169324b842 2023-02-12 15:00:17,876 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214010866 type: DISABLED version: 2 ttl: 0 2023-02-12 15:00:17,878 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214017877"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:17,878 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214017877"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:17,878 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214017877"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:17,878 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214017877"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:17,878 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214017877"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:17,878 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214017877"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:17,882 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-12 15:00:17,882 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-12 15:00:17,882 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214017878"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214017878"}]},"ts":"1676214017878"} 2023-02-12 15:00:17,882 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214017878"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214017878"}]},"ts":"1676214017878"} 2023-02-12 15:00:17,883 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214017878"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214017878"}]},"ts":"1676214017878"} 2023-02-12 15:00:17,883 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214017878"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214017878"}]},"ts":"1676214017878"} 2023-02-12 15:00:17,883 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214017878"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214017878"}]},"ts":"1676214017878"} 2023-02-12 15:00:17,883 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214017878"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214017878"}]},"ts":"1676214017878"} 2023-02-12 15:00:17,885 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-12 15:00:17,885 INFO [PEWorker-4] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-12 15:00:17,885 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-12 15:00:17,886 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676214007285 completed! 2023-02-12 15:00:17,888 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676214007285 snapshot=name: "snaptb0-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214010866 type: DISABLED version: 2 ttl: 0 ) in 76 msec 2023-02-12 15:00:18,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-12 15:00:18,075 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676214007285, procId: 91 completed 2023-02-12 15:00:18,075 INFO [Listener at localhost/38323] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676214007285-1676214017527 2023-02-12 15:00:18,080 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb0-1676214007285-1676214017527" 2023-02-12 15:00:18,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676214007285-1676214017527 2023-02-12 15:00:18,084 INFO [Listener at localhost/38323] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676214007285 2023-02-12 15:00:18,084 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676214007285 2023-02-12 15:00:18,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:18,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-12 15:00:18,088 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676214007285 2023-02-12 15:00:18,088 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214018088"}]},"ts":"1676214018088"} 2023-02-12 15:00:18,089 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLING in hbase:meta 2023-02-12 15:00:18,092 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 15:00:18,094 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN}] 2023-02-12 15:00:18,095 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN 2023-02-12 15:00:18,095 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN 2023-02-12 15:00:18,095 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN 2023-02-12 15:00:18,095 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN 2023-02-12 15:00:18,096 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:18,096 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN 2023-02-12 15:00:18,096 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:18,096 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:18,096 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:18,097 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN 2023-02-12 15:00:18,097 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:18,098 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:18,246 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-12 15:00:18,247 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 15:00:18,247 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 15:00:18,247 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 15:00:18,247 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 15:00:18,250 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:18,250 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:18,250 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:18,250 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:18,250 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:18,250 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214018250"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214018250"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214018250"}]},"ts":"1676214018250"} 2023-02-12 15:00:18,250 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214018250"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214018250"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214018250"}]},"ts":"1676214018250"} 2023-02-12 15:00:18,250 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214018250"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214018250"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214018250"}]},"ts":"1676214018250"} 2023-02-12 15:00:18,250 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214018250"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214018250"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214018250"}]},"ts":"1676214018250"} 2023-02-12 15:00:18,250 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214018250"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214018250"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214018250"}]},"ts":"1676214018250"} 2023-02-12 15:00:18,252 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=98, state=RUNNABLE; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:18,253 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=97, state=RUNNABLE; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:18,253 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=93, state=RUNNABLE; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:18,255 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=96, state=RUNNABLE; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:18,256 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=94, state=RUNNABLE; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:18,256 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:18,257 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214018256"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214018256"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214018256"}]},"ts":"1676214018256"} 2023-02-12 15:00:18,259 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=95, state=RUNNABLE; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:18,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-12 15:00:18,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:18,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 15:00:18,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:18,409 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:18,409 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:18,409 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:18,410 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:18,410 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:18,410 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:18,410 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:18,410 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:18,411 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:18,411 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 87a15dc17c09c51160706b761e764ba2 columnFamilyName cf 2023-02-12 15:00:18,411 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c70c08e0263941ae1bfbb2d57f1c3052 columnFamilyName cf 2023-02-12 15:00:18,411 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e31ecbd7855a2e0bca06cbe75ecfc8fe columnFamilyName cf 2023-02-12 15:00:18,418 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015 2023-02-12 15:00:18,418 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55 2023-02-12 15:00:18,418 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(310): Store=e31ecbd7855a2e0bca06cbe75ecfc8fe/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:18,418 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(310): Store=87a15dc17c09c51160706b761e764ba2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:18,418 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2 2023-02-12 15:00:18,419 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(310): Store=c70c08e0263941ae1bfbb2d57f1c3052/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:18,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:18,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:18,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:18,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:18,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:18,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:18,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:18,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:18,424 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e31ecbd7855a2e0bca06cbe75ecfc8fe; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68607780, jitterRate=0.022335588932037354}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:18,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Running coprocessor pre-open hook at 1676214018409Writing region info on filesystem at 1676214018409Initializing all the Stores at 1676214018409Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214018409Cleaning up temporary data from old regions at 1676214018421 (+12 ms)Cleaning up detritus from prior splits at 1676214018421Running coprocessor post-open hooks at 1676214018424 (+3 ms)Region opened successfully at 1676214018424 2023-02-12 15:00:18,425 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 87a15dc17c09c51160706b761e764ba2; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60895796, jitterRate=-0.09258192777633667}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:18,425 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 87a15dc17c09c51160706b761e764ba2: Running coprocessor pre-open hook at 1676214018409Writing region info on filesystem at 1676214018409Initializing all the Stores at 1676214018409Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214018409Cleaning up temporary data from old regions at 1676214018421 (+12 ms)Cleaning up detritus from prior splits at 1676214018421Running coprocessor post-open hooks at 1676214018425 (+4 ms)Region opened successfully at 1676214018425 2023-02-12 15:00:18,426 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2., pid=102, masterSystemTime=1676214018405 2023-02-12 15:00:18,426 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe., pid=103, masterSystemTime=1676214018405 2023-02-12 15:00:18,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:18,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c70c08e0263941ae1bfbb2d57f1c3052; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71933815, jitterRate=0.07189737260341644}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:18,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c70c08e0263941ae1bfbb2d57f1c3052: Running coprocessor pre-open hook at 1676214018408Writing region info on filesystem at 1676214018408Initializing all the Stores at 1676214018409 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214018409Cleaning up temporary data from old regions at 1676214018424 (+15 ms)Cleaning up detritus from prior splits at 1676214018424Running coprocessor post-open hooks at 1676214018427 (+3 ms)Region opened successfully at 1676214018427 2023-02-12 15:00:18,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:18,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:18,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:18,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 15:00:18,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:18,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052., pid=104, masterSystemTime=1676214018405 2023-02-12 15:00:18,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:18,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:18,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:18,428 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:18,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:18,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:18,428 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214018428"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214018428"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214018428"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214018428"}]},"ts":"1676214018428"} 2023-02-12 15:00:18,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:18,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 15:00:18,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:18,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:18,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:18,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:18,431 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:18,432 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:18,433 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:18,433 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 469de65a1b57113b3e7565d5f5bf8201 columnFamilyName cf 2023-02-12 15:00:18,434 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:18,434 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214018434"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214018434"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214018434"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214018434"}]},"ts":"1676214018434"} 2023-02-12 15:00:18,434 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:18,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:18,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:18,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:18,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 15:00:18,435 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:18,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:18,435 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214018435"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214018435"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214018435"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214018435"}]},"ts":"1676214018435"} 2023-02-12 15:00:18,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:18,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:18,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:18,437 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:18,438 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=94 2023-02-12 15:00:18,438 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=94, state=SUCCESS; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,38453,1676214004461 in 176 msec 2023-02-12 15:00:18,438 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:18,439 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=96 2023-02-12 15:00:18,439 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a2b878baabc37b3076d648a696d2d33f columnFamilyName cf 2023-02-12 15:00:18,439 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=96, state=SUCCESS; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,37113,1676214004377 in 181 msec 2023-02-12 15:00:18,440 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN in 346 msec 2023-02-12 15:00:18,441 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN in 345 msec 2023-02-12 15:00:18,441 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6 2023-02-12 15:00:18,441 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=95 2023-02-12 15:00:18,441 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=95, state=SUCCESS; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,33845,1676214004423 in 179 msec 2023-02-12 15:00:18,441 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:18,442 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN in 347 msec 2023-02-12 15:00:18,441 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(310): Store=469de65a1b57113b3e7565d5f5bf8201/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:18,444 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:18,444 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:18,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:18,444 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e116768b6cb1e8fb57dfa7c68914d39c columnFamilyName cf 2023-02-12 15:00:18,445 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d 2023-02-12 15:00:18,445 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(310): Store=a2b878baabc37b3076d648a696d2d33f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:18,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:18,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:18,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:18,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:18,451 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4 2023-02-12 15:00:18,451 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(310): Store=e116768b6cb1e8fb57dfa7c68914d39c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:18,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:18,453 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a2b878baabc37b3076d648a696d2d33f; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66498133, jitterRate=-0.009100601077079773}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:18,453 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a2b878baabc37b3076d648a696d2d33f: Running coprocessor pre-open hook at 1676214018429Writing region info on filesystem at 1676214018429Initializing all the Stores at 1676214018434 (+5 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214018434Cleaning up temporary data from old regions at 1676214018449 (+15 ms)Cleaning up detritus from prior splits at 1676214018449Running coprocessor post-open hooks at 1676214018453 (+4 ms)Region opened successfully at 1676214018453 2023-02-12 15:00:18,454 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f., pid=100, masterSystemTime=1676214018405 2023-02-12 15:00:18,454 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:18,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:18,455 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:18,456 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:18,456 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:18,456 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214018456"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214018456"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214018456"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214018456"}]},"ts":"1676214018456"} 2023-02-12 15:00:18,457 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 469de65a1b57113b3e7565d5f5bf8201; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64014468, jitterRate=-0.04611009359359741}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:18,458 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 469de65a1b57113b3e7565d5f5bf8201: Running coprocessor pre-open hook at 1676214018430Writing region info on filesystem at 1676214018430Initializing all the Stores at 1676214018431 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214018431Cleaning up temporary data from old regions at 1676214018448 (+17 ms)Cleaning up detritus from prior splits at 1676214018448Running coprocessor post-open hooks at 1676214018458 (+10 ms)Region opened successfully at 1676214018458 2023-02-12 15:00:18,458 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:18,459 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201., pid=101, masterSystemTime=1676214018405 2023-02-12 15:00:18,459 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e116768b6cb1e8fb57dfa7c68914d39c; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62899821, jitterRate=-0.06271962821483612}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:18,459 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e116768b6cb1e8fb57dfa7c68914d39c: Running coprocessor pre-open hook at 1676214018436Writing region info on filesystem at 1676214018436Initializing all the Stores at 1676214018440 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214018441 (+1 ms)Cleaning up temporary data from old regions at 1676214018454 (+13 ms)Cleaning up detritus from prior splits at 1676214018455 (+1 ms)Running coprocessor post-open hooks at 1676214018459 (+4 ms)Region opened successfully at 1676214018459 2023-02-12 15:00:18,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c., pid=99, masterSystemTime=1676214018405 2023-02-12 15:00:18,460 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=97 2023-02-12 15:00:18,460 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=97, state=SUCCESS; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,38453,1676214004461 in 205 msec 2023-02-12 15:00:18,460 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:18,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:18,461 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:18,461 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214018461"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214018461"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214018461"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214018461"}]},"ts":"1676214018461"} 2023-02-12 15:00:18,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:18,461 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:18,462 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN in 366 msec 2023-02-12 15:00:18,462 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:18,462 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214018461"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214018461"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214018461"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214018461"}]},"ts":"1676214018461"} 2023-02-12 15:00:18,464 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=93 2023-02-12 15:00:18,464 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=93, state=SUCCESS; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,37113,1676214004377 in 209 msec 2023-02-12 15:00:18,465 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=98 2023-02-12 15:00:18,465 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=98, state=SUCCESS; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423 in 211 msec 2023-02-12 15:00:18,465 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN in 372 msec 2023-02-12 15:00:18,466 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=98, resume processing ppid=92 2023-02-12 15:00:18,466 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN in 371 msec 2023-02-12 15:00:18,467 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214018467"}]},"ts":"1676214018467"} 2023-02-12 15:00:18,468 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLED in hbase:meta 2023-02-12 15:00:18,470 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676214007285' was successfully enabled. 2023-02-12 15:00:18,472 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676214007285 in 387 msec 2023-02-12 15:00:18,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-12 15:00:18,839 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 92 completed 2023-02-12 15:00:18,840 DEBUG [Listener at localhost/38323] zookeeper.ReadOnlyZKClient(139): Connect 0x0aebe9d0 to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:18,845 DEBUG [Listener at localhost/38323] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@272618c8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:18,847 DEBUG [hconnection-0x251148a9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:18,849 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43822, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:18,956 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 15:00:18,959 DEBUG [hconnection-0x251148a9-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:18,964 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59110, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:19,074 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 15:00:19,077 DEBUG [hconnection-0x251148a9-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:19,080 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55994, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:19,168 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 15:00:19,278 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 15:00:19,362 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 15:00:20,231 INFO [Listener at localhost/38323] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676214007285 2023-02-12 15:00:20,232 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676214007285 2023-02-12 15:00:20,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:20,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-12 15:00:20,237 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214020237"}]},"ts":"1676214020237"} 2023-02-12 15:00:20,238 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLING in hbase:meta 2023-02-12 15:00:20,241 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676214007285 to state=DISABLING 2023-02-12 15:00:20,242 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN}] 2023-02-12 15:00:20,243 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN 2023-02-12 15:00:20,243 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN 2023-02-12 15:00:20,243 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN 2023-02-12 15:00:20,244 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN 2023-02-12 15:00:20,244 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN 2023-02-12 15:00:20,244 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:20,244 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214020244"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214020244"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214020244"}]},"ts":"1676214020244"} 2023-02-12 15:00:20,244 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:20,245 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:20,245 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214020244"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214020244"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214020244"}]},"ts":"1676214020244"} 2023-02-12 15:00:20,245 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214020244"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214020244"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214020244"}]},"ts":"1676214020244"} 2023-02-12 15:00:20,247 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:20,247 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:20,247 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214020246"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214020246"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214020246"}]},"ts":"1676214020246"} 2023-02-12 15:00:20,247 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214020246"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214020246"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214020246"}]},"ts":"1676214020246"} 2023-02-12 15:00:20,248 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=111, state=RUNNABLE; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:20,251 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=109, state=RUNNABLE; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:20,252 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=110, state=RUNNABLE; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:20,253 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=107, state=RUNNABLE; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:20,254 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=108, state=RUNNABLE; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:20,255 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN 2023-02-12 15:00:20,257 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:20,257 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214020256"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214020256"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214020256"}]},"ts":"1676214020256"} 2023-02-12 15:00:20,258 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=106, state=RUNNABLE; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:20,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:20,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e116768b6cb1e8fb57dfa7c68914d39c, disabling compactions & flushes 2023-02-12 15:00:20,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:20,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:20,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. after waiting 0 ms 2023-02-12 15:00:20,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:20,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:20,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:20,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a2b878baabc37b3076d648a696d2d33f, disabling compactions & flushes 2023-02-12 15:00:20,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 87a15dc17c09c51160706b761e764ba2, disabling compactions & flushes 2023-02-12 15:00:20,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:20,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:20,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:20,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:20,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. after waiting 0 ms 2023-02-12 15:00:20,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. after waiting 0 ms 2023-02-12 15:00:20,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:20,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:20,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-12 15:00:20,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-12 15:00:20,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-12 15:00:20,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:20,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:20,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e116768b6cb1e8fb57dfa7c68914d39c: Waiting for close lock at 1676214020404Running coprocessor pre-close hooks at 1676214020404Disabling compacts and flushes for region at 1676214020404Disabling writes for close at 1676214020405 (+1 ms)Writing region close event to WAL at 1676214020409 (+4 ms)Running coprocessor post-close hooks at 1676214020418 (+9 ms)Closed at 1676214020418 2023-02-12 15:00:20,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a2b878baabc37b3076d648a696d2d33f: Waiting for close lock at 1676214020405Running coprocessor pre-close hooks at 1676214020405Disabling compacts and flushes for region at 1676214020405Disabling writes for close at 1676214020409 (+4 ms)Writing region close event to WAL at 1676214020415 (+6 ms)Running coprocessor post-close hooks at 1676214020418 (+3 ms)Closed at 1676214020418 2023-02-12 15:00:20,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:20,418 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 87a15dc17c09c51160706b761e764ba2: Waiting for close lock at 1676214020405Running coprocessor pre-close hooks at 1676214020405Disabling compacts and flushes for region at 1676214020405Disabling writes for close at 1676214020409 (+4 ms)Writing region close event to WAL at 1676214020415 (+6 ms)Running coprocessor post-close hooks at 1676214020418 (+3 ms)Closed at 1676214020418 2023-02-12 15:00:20,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:20,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:20,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c70c08e0263941ae1bfbb2d57f1c3052, disabling compactions & flushes 2023-02-12 15:00:20,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:20,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:20,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. after waiting 0 ms 2023-02-12 15:00:20,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:20,420 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSED 2023-02-12 15:00:20,420 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214020420"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214020420"}]},"ts":"1676214020420"} 2023-02-12 15:00:20,421 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSED 2023-02-12 15:00:20,421 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214020421"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214020421"}]},"ts":"1676214020421"} 2023-02-12 15:00:20,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:20,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:20,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 469de65a1b57113b3e7565d5f5bf8201, disabling compactions & flushes 2023-02-12 15:00:20,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:20,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:20,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. after waiting 0 ms 2023-02-12 15:00:20,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:20,422 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSED 2023-02-12 15:00:20,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:20,422 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214020422"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214020422"}]},"ts":"1676214020422"} 2023-02-12 15:00:20,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:20,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e31ecbd7855a2e0bca06cbe75ecfc8fe, disabling compactions & flushes 2023-02-12 15:00:20,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:20,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:20,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. after waiting 0 ms 2023-02-12 15:00:20,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:20,426 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=111 2023-02-12 15:00:20,426 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=111, state=SUCCESS; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423 in 175 msec 2023-02-12 15:00:20,427 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=110 2023-02-12 15:00:20,427 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=110, state=SUCCESS; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,38453,1676214004461 in 171 msec 2023-02-12 15:00:20,428 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=109 2023-02-12 15:00:20,428 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=109, state=SUCCESS; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,37113,1676214004377 in 176 msec 2023-02-12 15:00:20,428 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN in 184 msec 2023-02-12 15:00:20,428 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN in 185 msec 2023-02-12 15:00:20,429 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-12 15:00:20,429 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-12 15:00:20,429 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:20,429 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-12 15:00:20,429 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 469de65a1b57113b3e7565d5f5bf8201: Waiting for close lock at 1676214020422Running coprocessor pre-close hooks at 1676214020422Disabling compacts and flushes for region at 1676214020422Disabling writes for close at 1676214020422Writing region close event to WAL at 1676214020423 (+1 ms)Running coprocessor post-close hooks at 1676214020429 (+6 ms)Closed at 1676214020429 2023-02-12 15:00:20,429 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:20,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN in 186 msec 2023-02-12 15:00:20,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c70c08e0263941ae1bfbb2d57f1c3052: Waiting for close lock at 1676214020420Running coprocessor pre-close hooks at 1676214020420Disabling compacts and flushes for region at 1676214020420Disabling writes for close at 1676214020420Writing region close event to WAL at 1676214020421 (+1 ms)Running coprocessor post-close hooks at 1676214020429 (+8 ms)Closed at 1676214020429 2023-02-12 15:00:20,430 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:20,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Waiting for close lock at 1676214020423Running coprocessor pre-close hooks at 1676214020423Disabling compacts and flushes for region at 1676214020423Disabling writes for close at 1676214020423Writing region close event to WAL at 1676214020424 (+1 ms)Running coprocessor post-close hooks at 1676214020430 (+6 ms)Closed at 1676214020430 2023-02-12 15:00:20,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:20,431 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSED 2023-02-12 15:00:20,431 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214020431"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214020431"}]},"ts":"1676214020431"} 2023-02-12 15:00:20,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:20,432 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSED 2023-02-12 15:00:20,432 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214020432"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214020432"}]},"ts":"1676214020432"} 2023-02-12 15:00:20,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:20,433 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSED 2023-02-12 15:00:20,433 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214020433"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214020433"}]},"ts":"1676214020433"} 2023-02-12 15:00:20,434 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=106 2023-02-12 15:00:20,434 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=106, state=SUCCESS; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,37113,1676214004377 in 175 msec 2023-02-12 15:00:20,435 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=108 2023-02-12 15:00:20,435 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=108, state=SUCCESS; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,33845,1676214004423 in 179 msec 2023-02-12 15:00:20,436 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN in 192 msec 2023-02-12 15:00:20,436 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN in 193 msec 2023-02-12 15:00:20,436 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=107 2023-02-12 15:00:20,436 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=107, state=SUCCESS; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,38453,1676214004461 in 182 msec 2023-02-12 15:00:20,438 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=107, resume processing ppid=105 2023-02-12 15:00:20,438 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN in 194 msec 2023-02-12 15:00:20,439 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214020439"}]},"ts":"1676214020439"} 2023-02-12 15:00:20,440 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLED in hbase:meta 2023-02-12 15:00:20,442 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676214007285 to state=DISABLED 2023-02-12 15:00:20,443 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676214007285 in 210 msec 2023-02-12 15:00:20,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-12 15:00:20,488 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 105 completed 2023-02-12 15:00:20,496 INFO [Listener at localhost/38323] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676214007285-1676214020496 2023-02-12 15:00:20,497 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676214007285-1676214020496 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:20,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676214020497 (current time:1676214020497). 2023-02-12 15:00:20,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 15:00:20,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(139): Connect 0x41f4d6ce to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:20,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3703cfa2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:20,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:20,506 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43824, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:20,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41f4d6ce to 127.0.0.1:61893 2023-02-12 15:00:20,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:20,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 15:00:20,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676214007285-1676214020496 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:20,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:20,510 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:20,511 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE; timeout=600000, timestamp=1676214620511 2023-02-12 15:00:20,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676214007285-1676214020496 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:20,511 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676214007285-1676214020496 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676214007285 2023-02-12 15:00:20,511 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676214007285-1676214020496 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 15:00:20,511 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 15:00:20,520 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676214007285-1676214020496 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } 2023-02-12 15:00:20,521 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:20,521 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:20,522 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:20,522 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:20,522 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:20,522 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:20,522 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:20,522 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:20,522 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:20,522 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:20,522 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:20,522 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:20,522 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:20,522 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:20,523 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015] hfiles 2023-02-12 15:00:20,524 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015 2023-02-12 15:00:20,524 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d] hfiles 2023-02-12 15:00:20,524 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d 2023-02-12 15:00:20,525 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6] hfiles 2023-02-12 15:00:20,525 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6 2023-02-12 15:00:20,525 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2] hfiles 2023-02-12 15:00:20,525 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2 2023-02-12 15:00:20,525 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4] hfiles 2023-02-12 15:00:20,525 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4 2023-02-12 15:00:20,526 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55] hfiles 2023-02-12 15:00:20,526 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55 2023-02-12 15:00:20,531 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766] hfiles 2023-02-12 15:00:20,531 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 2023-02-12 15:00:20,531 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 2023-02-12 15:00:20,531 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c 2023-02-12 15:00:20,531 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b 2023-02-12 15:00:20,531 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a 2023-02-12 15:00:20,531 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 2023-02-12 15:00:20,761 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 15:00:20,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676214007285-1676214020496 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } is done 2023-02-12 15:00:20,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(404): Snapshoting '{ ss=hbase-failsafe-emptySnaptb-1676214007285-1676214020496 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }' is still in progress! 2023-02-12 15:00:20,763 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-12 15:00:20,951 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676214007285-1676214020496 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } as finished. 2023-02-12 15:00:20,952 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676214007285-1676214020496 2023-02-12 15:00:20,952 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676214007285-1676214020496 2023-02-12 15:00:20,972 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676214007285-1676214020496 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676214007285-1676214020496 2023-02-12 15:00:20,991 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676214007285-1676214020496 of table testRestoreSnapshot-1676214007285 completed 2023-02-12 15:00:20,991 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676214007285-1676214020496 2023-02-12 15:00:20,992 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676214007285-1676214020496 2023-02-12 15:00:20,992 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676214007285-1676214020496 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676214007285 at 1676214020511Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676214007285-1676214020496 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } at 1676214020520 (+9 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676214007285-1676214020496 at 1676214020952 (+432 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676214007285-1676214020496 at 1676214020952Writing data manifest for hbase-failsafe-emptySnaptb-1676214007285-1676214020496 at 1676214020955 (+3 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676214007285-1676214020496 at 1676214020965 (+10 ms)Snapshot hbase-failsafe-emptySnaptb-1676214007285-1676214020496 of table testRestoreSnapshot-1676214007285 completed at 1676214020991 (+26 ms) 2023-02-12 15:00:20,993 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:20,995 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE in 484 msec 2023-02-12 15:00:21,263 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 15:00:21,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676214007285-1676214020496 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } is done 2023-02-12 15:00:21,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676214007285-1676214020496 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 15:00:21,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676214007285 snapshot=name: "emptySnaptb-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214008430 type: DISABLED version: 2 ttl: 0 ) 2023-02-12 15:00:21,270 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676214007285 as table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:21,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-12 15:00:21,282 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/.tabledesc/.tableinfo.0000000003 2023-02-12 15:00:21,283 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/.tabledesc/.tableinfo.0000000002 2023-02-12 15:00:21,283 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/.tabledesc/.tableinfo.0000000003 2023-02-12 15:00:21,284 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } 2023-02-12 15:00:21,285 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214008430 type: DISABLED version: 2 ttl: 0 2023-02-12 15:00:21,285 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285 2023-02-12 15:00:21,292 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:21,295 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 2023-02-12 15:00:21,296 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 2023-02-12 15:00:21,297 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c 2023-02-12 15:00:21,298 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b 2023-02-12 15:00:21,300 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a 2023-02-12 15:00:21,301 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 2023-02-12 15:00:21,301 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:21,301 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:21,301 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:21,301 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:21,301 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:21,301 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:21,307 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/f5c092a8a2e84ffe91bb687f3ea1c42d 2023-02-12 15:00:21,307 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/e0edae35a0ff4d6db2a54417a26dc5a6 2023-02-12 15:00:21,308 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/5718cdbb97f24b3081185dfde2fb1e55 2023-02-12 15:00:21,308 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/498d9c0eb3304fc7aedc40e7676473e2 2023-02-12 15:00:21,310 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/4a211d423a9d40e08d63ef96097f6015 2023-02-12 15:00:21,311 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/b00b668a6b014eb4a4e4c4bb861069f4 2023-02-12 15:00:21,311 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214008430 type: DISABLED version: 2 ttl: 0 2023-02-12 15:00:21,312 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214021312"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:21,312 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214021312"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:21,312 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214021312"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:21,313 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214021312"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:21,313 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214021312"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:21,313 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214021312"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:21,315 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-12 15:00:21,315 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-12 15:00:21,315 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214021313"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214021313"}]},"ts":"1676214021313"} 2023-02-12 15:00:21,315 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214021313"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214021313"}]},"ts":"1676214021313"} 2023-02-12 15:00:21,315 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214021313"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214021313"}]},"ts":"1676214021313"} 2023-02-12 15:00:21,315 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214021313"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214021313"}]},"ts":"1676214021313"} 2023-02-12 15:00:21,315 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214021313"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214021313"}]},"ts":"1676214021313"} 2023-02-12 15:00:21,315 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214021313"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214021313"}]},"ts":"1676214021313"} 2023-02-12 15:00:21,317 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-12 15:00:21,317 INFO [PEWorker-3] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-12 15:00:21,317 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-12 15:00:21,318 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676214007285 completed! 2023-02-12 15:00:21,319 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676214007285 snapshot=name: "emptySnaptb-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214008430 type: DISABLED version: 2 ttl: 0 ) in 49 msec 2023-02-12 15:00:21,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-12 15:00:21,524 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676214007285, procId: 119 completed 2023-02-12 15:00:21,524 INFO [Listener at localhost/38323] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676214007285-1676214020496 2023-02-12 15:00:21,524 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-emptySnaptb-1676214007285-1676214020496" 2023-02-12 15:00:21,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676214007285-1676214020496 2023-02-12 15:00:21,527 INFO [Listener at localhost/38323] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676214007285 2023-02-12 15:00:21,528 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676214007285 2023-02-12 15:00:21,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:21,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-12 15:00:21,531 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676214007285 2023-02-12 15:00:21,531 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214021531"}]},"ts":"1676214021531"} 2023-02-12 15:00:21,532 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLING in hbase:meta 2023-02-12 15:00:21,536 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 15:00:21,536 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN}] 2023-02-12 15:00:21,537 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN 2023-02-12 15:00:21,537 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN 2023-02-12 15:00:21,538 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN 2023-02-12 15:00:21,538 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN 2023-02-12 15:00:21,538 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:21,538 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:21,538 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:21,539 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:21,539 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN 2023-02-12 15:00:21,539 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN 2023-02-12 15:00:21,540 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:21,540 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:21,689 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-12 15:00:21,689 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 15:00:21,689 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 15:00:21,689 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 15:00:21,689 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 15:00:21,694 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:21,694 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:21,694 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:21,694 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:21,694 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214021694"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214021694"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214021694"}]},"ts":"1676214021694"} 2023-02-12 15:00:21,694 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:21,694 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214021694"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214021694"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214021694"}]},"ts":"1676214021694"} 2023-02-12 15:00:21,694 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214021694"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214021694"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214021694"}]},"ts":"1676214021694"} 2023-02-12 15:00:21,694 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214021694"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214021694"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214021694"}]},"ts":"1676214021694"} 2023-02-12 15:00:21,694 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214021694"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214021694"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214021694"}]},"ts":"1676214021694"} 2023-02-12 15:00:21,696 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=122, state=RUNNABLE; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:21,696 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=125, state=RUNNABLE; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:21,697 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=126, state=RUNNABLE; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:21,698 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=124, state=RUNNABLE; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:21,699 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=121, state=RUNNABLE; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:21,700 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:21,700 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214021700"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214021700"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214021700"}]},"ts":"1676214021700"} 2023-02-12 15:00:21,703 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=123, state=RUNNABLE; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:21,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-12 15:00:21,850 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:21,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 15:00:21,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:21,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:21,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 15:00:21,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:21,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:21,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:21,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:21,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:21,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:21,851 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:21,852 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:21,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 15:00:21,852 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:21,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:21,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:21,852 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:21,852 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:21,853 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:21,854 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:21,854 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:21,854 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:21,854 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:21,854 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:21,854 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a2b878baabc37b3076d648a696d2d33f columnFamilyName cf 2023-02-12 15:00:21,854 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e116768b6cb1e8fb57dfa7c68914d39c columnFamilyName cf 2023-02-12 15:00:21,855 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:21,855 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:21,855 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(310): Store=a2b878baabc37b3076d648a696d2d33f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:21,855 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(310): Store=e116768b6cb1e8fb57dfa7c68914d39c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:21,855 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 87a15dc17c09c51160706b761e764ba2 columnFamilyName cf 2023-02-12 15:00:21,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:21,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:21,856 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(310): Store=87a15dc17c09c51160706b761e764ba2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:21,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:21,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:21,857 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:21,858 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:21,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:21,859 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:21,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:21,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a2b878baabc37b3076d648a696d2d33f; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67475025, jitterRate=0.005456224083900452}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:21,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a2b878baabc37b3076d648a696d2d33f: Running coprocessor pre-open hook at 1676214021851Writing region info on filesystem at 1676214021851Initializing all the Stores at 1676214021852 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214021852Cleaning up temporary data from old regions at 1676214021857 (+5 ms)Cleaning up detritus from prior splits at 1676214021858 (+1 ms)Running coprocessor post-open hooks at 1676214021860 (+2 ms)Region opened successfully at 1676214021860 2023-02-12 15:00:21,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e116768b6cb1e8fb57dfa7c68914d39c; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75304090, jitterRate=0.12211838364601135}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:21,860 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e116768b6cb1e8fb57dfa7c68914d39c: Running coprocessor pre-open hook at 1676214021851Writing region info on filesystem at 1676214021851Initializing all the Stores at 1676214021852 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214021852Cleaning up temporary data from old regions at 1676214021858 (+6 ms)Cleaning up detritus from prior splits at 1676214021858Running coprocessor post-open hooks at 1676214021860 (+2 ms)Region opened successfully at 1676214021860 2023-02-12 15:00:21,861 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 87a15dc17c09c51160706b761e764ba2; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70743027, jitterRate=0.05415324866771698}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:21,861 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 87a15dc17c09c51160706b761e764ba2: Running coprocessor pre-open hook at 1676214021853Writing region info on filesystem at 1676214021853Initializing all the Stores at 1676214021853Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214021853Cleaning up temporary data from old regions at 1676214021858 (+5 ms)Cleaning up detritus from prior splits at 1676214021858Running coprocessor post-open hooks at 1676214021861 (+3 ms)Region opened successfully at 1676214021861 2023-02-12 15:00:21,861 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f., pid=128, masterSystemTime=1676214021848 2023-02-12 15:00:21,861 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c., pid=129, masterSystemTime=1676214021849 2023-02-12 15:00:21,861 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2., pid=130, masterSystemTime=1676214021850 2023-02-12 15:00:21,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:21,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:21,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:21,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 15:00:21,863 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:21,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:21,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:21,863 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214021862"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214021862"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214021862"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214021862"}]},"ts":"1676214021862"} 2023-02-12 15:00:21,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:21,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:21,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:21,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:21,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:21,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 15:00:21,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:21,864 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:21,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:21,864 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214021864"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214021864"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214021864"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214021864"}]},"ts":"1676214021864"} 2023-02-12 15:00:21,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:21,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:21,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:21,864 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:21,864 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:21,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 15:00:21,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:21,865 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:21,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:21,865 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214021865"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214021865"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214021865"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214021865"}]},"ts":"1676214021865"} 2023-02-12 15:00:21,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:21,865 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:21,867 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=125 2023-02-12 15:00:21,867 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=125, state=SUCCESS; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,38453,1676214004461 in 169 msec 2023-02-12 15:00:21,867 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-12 15:00:21,868 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=126 2023-02-12 15:00:21,868 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=126, state=SUCCESS; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423 in 169 msec 2023-02-12 15:00:21,870 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN in 331 msec 2023-02-12 15:00:21,870 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN in 332 msec 2023-02-12 15:00:21,871 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=124 2023-02-12 15:00:21,871 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=124, state=SUCCESS; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,37113,1676214004377 in 170 msec 2023-02-12 15:00:21,872 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN in 335 msec 2023-02-12 15:00:21,872 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:21,876 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:21,877 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:21,877 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:21,877 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e31ecbd7855a2e0bca06cbe75ecfc8fe columnFamilyName cf 2023-02-12 15:00:21,878 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(310): Store=e31ecbd7855a2e0bca06cbe75ecfc8fe/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:21,882 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:21,883 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:21,883 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:21,883 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:21,884 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c70c08e0263941ae1bfbb2d57f1c3052 columnFamilyName cf 2023-02-12 15:00:21,884 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:21,884 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:21,884 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 469de65a1b57113b3e7565d5f5bf8201 columnFamilyName cf 2023-02-12 15:00:21,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:21,884 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(310): Store=c70c08e0263941ae1bfbb2d57f1c3052/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:21,885 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(310): Store=469de65a1b57113b3e7565d5f5bf8201/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:21,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:21,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:21,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:21,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:21,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:21,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e31ecbd7855a2e0bca06cbe75ecfc8fe; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66020071, jitterRate=-0.01622427999973297}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:21,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Running coprocessor pre-open hook at 1676214021863Writing region info on filesystem at 1676214021863Initializing all the Stores at 1676214021869 (+6 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214021869Cleaning up temporary data from old regions at 1676214021885 (+16 ms)Cleaning up detritus from prior splits at 1676214021885Running coprocessor post-open hooks at 1676214021888 (+3 ms)Region opened successfully at 1676214021888 2023-02-12 15:00:21,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:21,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe., pid=127, masterSystemTime=1676214021848 2023-02-12 15:00:21,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:21,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c70c08e0263941ae1bfbb2d57f1c3052; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70253007, jitterRate=0.04685138165950775}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:21,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c70c08e0263941ae1bfbb2d57f1c3052: Running coprocessor pre-open hook at 1676214021864Writing region info on filesystem at 1676214021864Initializing all the Stores at 1676214021869 (+5 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214021869Cleaning up temporary data from old regions at 1676214021887 (+18 ms)Cleaning up detritus from prior splits at 1676214021888 (+1 ms)Running coprocessor post-open hooks at 1676214021891 (+3 ms)Region opened successfully at 1676214021891 2023-02-12 15:00:21,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 469de65a1b57113b3e7565d5f5bf8201; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74696726, jitterRate=0.11306795477867126}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:21,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052., pid=132, masterSystemTime=1676214021849 2023-02-12 15:00:21,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 469de65a1b57113b3e7565d5f5bf8201: Running coprocessor pre-open hook at 1676214021865Writing region info on filesystem at 1676214021865Initializing all the Stores at 1676214021869 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214021869Cleaning up temporary data from old regions at 1676214021888 (+19 ms)Cleaning up detritus from prior splits at 1676214021889 (+1 ms)Running coprocessor post-open hooks at 1676214021892 (+3 ms)Region opened successfully at 1676214021892 2023-02-12 15:00:21,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:21,893 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:21,893 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201., pid=131, masterSystemTime=1676214021850 2023-02-12 15:00:21,894 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:21,894 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214021893"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214021893"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214021893"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214021893"}]},"ts":"1676214021893"} 2023-02-12 15:00:21,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:21,895 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:21,896 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:21,897 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214021896"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214021896"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214021896"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214021896"}]},"ts":"1676214021896"} 2023-02-12 15:00:21,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:21,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:21,897 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:21,897 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214021897"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214021897"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214021897"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214021897"}]},"ts":"1676214021897"} 2023-02-12 15:00:21,898 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=122 2023-02-12 15:00:21,898 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=122, state=SUCCESS; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,38453,1676214004461 in 200 msec 2023-02-12 15:00:21,900 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN in 362 msec 2023-02-12 15:00:21,901 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=123 2023-02-12 15:00:21,901 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=123, state=SUCCESS; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,33845,1676214004423 in 196 msec 2023-02-12 15:00:21,901 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=121 2023-02-12 15:00:21,901 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=121, state=SUCCESS; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,37113,1676214004377 in 201 msec 2023-02-12 15:00:21,902 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN in 365 msec 2023-02-12 15:00:21,903 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=121, resume processing ppid=120 2023-02-12 15:00:21,903 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN in 365 msec 2023-02-12 15:00:21,903 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214021903"}]},"ts":"1676214021903"} 2023-02-12 15:00:21,904 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLED in hbase:meta 2023-02-12 15:00:21,906 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676214007285' was successfully enabled. 2023-02-12 15:00:21,908 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676214007285 in 379 msec 2023-02-12 15:00:22,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-12 15:00:22,283 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 120 completed 2023-02-12 15:00:22,283 DEBUG [Listener at localhost/38323] zookeeper.ReadOnlyZKClient(139): Connect 0x6452cab0 to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:22,289 DEBUG [Listener at localhost/38323] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43ec0260, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:22,291 DEBUG [hconnection-0x2606f9a8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:22,293 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43840, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:22,297 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 15:00:22,300 DEBUG [hconnection-0x2606f9a8-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:22,303 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59120, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:22,304 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 15:00:22,308 DEBUG [hconnection-0x2606f9a8-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:22,309 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56010, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:22,310 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 15:00:22,314 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 15:00:22,320 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 15:00:22,327 INFO [Listener at localhost/38323] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,327 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-12 15:00:22,331 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214022331"}]},"ts":"1676214022331"} 2023-02-12 15:00:22,332 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLING in hbase:meta 2023-02-12 15:00:22,334 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676214007285 to state=DISABLING 2023-02-12 15:00:22,335 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN}] 2023-02-12 15:00:22,340 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN 2023-02-12 15:00:22,341 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN 2023-02-12 15:00:22,341 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN 2023-02-12 15:00:22,341 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN 2023-02-12 15:00:22,341 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN 2023-02-12 15:00:22,342 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:22,342 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:22,342 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214022342"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214022342"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214022342"}]},"ts":"1676214022342"} 2023-02-12 15:00:22,342 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:22,342 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214022342"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214022342"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214022342"}]},"ts":"1676214022342"} 2023-02-12 15:00:22,342 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214022342"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214022342"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214022342"}]},"ts":"1676214022342"} 2023-02-12 15:00:22,342 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:22,342 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:22,342 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214022342"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214022342"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214022342"}]},"ts":"1676214022342"} 2023-02-12 15:00:22,343 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214022342"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214022342"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214022342"}]},"ts":"1676214022342"} 2023-02-12 15:00:22,344 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=136, state=RUNNABLE; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:22,345 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=137, state=RUNNABLE; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:22,346 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=135, state=RUNNABLE; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:22,347 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=134, state=RUNNABLE; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:22,347 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN 2023-02-12 15:00:22,348 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=139, state=RUNNABLE; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:22,349 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:22,349 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214022349"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214022349"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214022349"}]},"ts":"1676214022349"} 2023-02-12 15:00:22,350 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=138, state=RUNNABLE; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:22,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:22,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e116768b6cb1e8fb57dfa7c68914d39c, disabling compactions & flushes 2023-02-12 15:00:22,497 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:22,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:22,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. after waiting 0 ms 2023-02-12 15:00:22,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:22,498 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:22,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 469de65a1b57113b3e7565d5f5bf8201, disabling compactions & flushes 2023-02-12 15:00:22,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:22,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:22,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. after waiting 0 ms 2023-02-12 15:00:22,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:22,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:22,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a2b878baabc37b3076d648a696d2d33f, disabling compactions & flushes 2023-02-12 15:00:22,500 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:22,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:22,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. after waiting 0 ms 2023-02-12 15:00:22,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:22,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-12 15:00:22,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-12 15:00:22,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:22,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e116768b6cb1e8fb57dfa7c68914d39c: Waiting for close lock at 1676214022496Running coprocessor pre-close hooks at 1676214022496Disabling compacts and flushes for region at 1676214022496Disabling writes for close at 1676214022497 (+1 ms)Writing region close event to WAL at 1676214022498 (+1 ms)Running coprocessor post-close hooks at 1676214022503 (+5 ms)Closed at 1676214022504 (+1 ms) 2023-02-12 15:00:22,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-12 15:00:22,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:22,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 469de65a1b57113b3e7565d5f5bf8201: Waiting for close lock at 1676214022498Running coprocessor pre-close hooks at 1676214022498Disabling compacts and flushes for region at 1676214022498Disabling writes for close at 1676214022499 (+1 ms)Writing region close event to WAL at 1676214022500 (+1 ms)Running coprocessor post-close hooks at 1676214022504 (+4 ms)Closed at 1676214022504 2023-02-12 15:00:22,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:22,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a2b878baabc37b3076d648a696d2d33f: Waiting for close lock at 1676214022499Running coprocessor pre-close hooks at 1676214022499Disabling compacts and flushes for region at 1676214022499Disabling writes for close at 1676214022500 (+1 ms)Writing region close event to WAL at 1676214022500Running coprocessor post-close hooks at 1676214022505 (+5 ms)Closed at 1676214022505 2023-02-12 15:00:22,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:22,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:22,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c70c08e0263941ae1bfbb2d57f1c3052, disabling compactions & flushes 2023-02-12 15:00:22,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:22,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:22,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. after waiting 0 ms 2023-02-12 15:00:22,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:22,508 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSED 2023-02-12 15:00:22,508 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214022507"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214022507"}]},"ts":"1676214022507"} 2023-02-12 15:00:22,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:22,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:22,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 87a15dc17c09c51160706b761e764ba2, disabling compactions & flushes 2023-02-12 15:00:22,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:22,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:22,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. after waiting 0 ms 2023-02-12 15:00:22,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:22,509 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSED 2023-02-12 15:00:22,509 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214022509"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214022509"}]},"ts":"1676214022509"} 2023-02-12 15:00:22,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:22,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:22,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e31ecbd7855a2e0bca06cbe75ecfc8fe, disabling compactions & flushes 2023-02-12 15:00:22,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:22,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:22,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. after waiting 0 ms 2023-02-12 15:00:22,511 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:22,511 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSED 2023-02-12 15:00:22,511 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214022511"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214022511"}]},"ts":"1676214022511"} 2023-02-12 15:00:22,512 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-12 15:00:22,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:22,513 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c70c08e0263941ae1bfbb2d57f1c3052: Waiting for close lock at 1676214022506Running coprocessor pre-close hooks at 1676214022506Disabling compacts and flushes for region at 1676214022506Disabling writes for close at 1676214022507 (+1 ms)Writing region close event to WAL at 1676214022507Running coprocessor post-close hooks at 1676214022512 (+5 ms)Closed at 1676214022513 (+1 ms) 2023-02-12 15:00:22,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-12 15:00:22,515 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:22,515 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 87a15dc17c09c51160706b761e764ba2: Waiting for close lock at 1676214022508Running coprocessor pre-close hooks at 1676214022508Disabling compacts and flushes for region at 1676214022508Disabling writes for close at 1676214022509 (+1 ms)Writing region close event to WAL at 1676214022509Running coprocessor post-close hooks at 1676214022515 (+6 ms)Closed at 1676214022515 2023-02-12 15:00:22,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-12 15:00:22,517 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:22,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:22,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Waiting for close lock at 1676214022509Running coprocessor pre-close hooks at 1676214022509Disabling compacts and flushes for region at 1676214022509Disabling writes for close at 1676214022511 (+2 ms)Writing region close event to WAL at 1676214022511Running coprocessor post-close hooks at 1676214022517 (+6 ms)Closed at 1676214022518 (+1 ms) 2023-02-12 15:00:22,518 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=139 2023-02-12 15:00:22,518 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSED 2023-02-12 15:00:22,518 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=139, state=SUCCESS; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423 in 163 msec 2023-02-12 15:00:22,518 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214022518"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214022518"}]},"ts":"1676214022518"} 2023-02-12 15:00:22,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:22,519 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=134 2023-02-12 15:00:22,520 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=134, state=SUCCESS; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,37113,1676214004377 in 166 msec 2023-02-12 15:00:22,522 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSED 2023-02-12 15:00:22,522 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN in 183 msec 2023-02-12 15:00:22,522 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214022522"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214022522"}]},"ts":"1676214022522"} 2023-02-12 15:00:22,524 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=138 2023-02-12 15:00:22,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:22,524 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=138, state=SUCCESS; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,38453,1676214004461 in 166 msec 2023-02-12 15:00:22,524 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSED 2023-02-12 15:00:22,524 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214022524"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214022524"}]},"ts":"1676214022524"} 2023-02-12 15:00:22,525 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN in 185 msec 2023-02-12 15:00:22,525 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=136 2023-02-12 15:00:22,526 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=136, state=SUCCESS; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,33845,1676214004423 in 176 msec 2023-02-12 15:00:22,526 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN in 189 msec 2023-02-12 15:00:22,527 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=137 2023-02-12 15:00:22,527 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=137, state=SUCCESS; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,37113,1676214004377 in 179 msec 2023-02-12 15:00:22,527 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN in 191 msec 2023-02-12 15:00:22,528 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=135 2023-02-12 15:00:22,528 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=135, state=SUCCESS; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,38453,1676214004461 in 180 msec 2023-02-12 15:00:22,530 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN in 192 msec 2023-02-12 15:00:22,531 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=135, resume processing ppid=133 2023-02-12 15:00:22,531 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN in 193 msec 2023-02-12 15:00:22,531 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214022531"}]},"ts":"1676214022531"} 2023-02-12 15:00:22,533 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLED in hbase:meta 2023-02-12 15:00:22,535 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676214007285 to state=DISABLED 2023-02-12 15:00:22,537 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676214007285 in 208 msec 2023-02-12 15:00:22,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-12 15:00:22,582 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 133 completed 2023-02-12 15:00:22,591 INFO [Listener at localhost/38323] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676214007285-1676214022591 2023-02-12 15:00:22,592 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676214007285-1676214022591 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:22,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676214022592 (current time:1676214022592). 2023-02-12 15:00:22,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 15:00:22,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(139): Connect 0x66ba622a to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:22,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49fd1e71, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:22,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:22,600 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43850, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:22,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ba622a to 127.0.0.1:61893 2023-02-12 15:00:22,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:22,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 15:00:22,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676214007285-1676214022591 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:22,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:22,604 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:22,605 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE; timeout=600000, timestamp=1676214622605 2023-02-12 15:00:22,605 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676214007285-1676214022591 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676214007285-1676214022591 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } 2023-02-12 15:00:22,605 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676214007285-1676214022591 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 15:00:22,605 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 15:00:22,614 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676214007285-1676214022591 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } 2023-02-12 15:00:22,617 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:22,617 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:22,617 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:22,617 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:22,617 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:22,617 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:22,617 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:22,617 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:22,618 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:22,618 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:22,618 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:22,618 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:22,618 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:22,618 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:22,618 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:22,619 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:22,619 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:22,619 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:22,619 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:22,619 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:22,619 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:22,645 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676214007285-1676214022591 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } as finished. 2023-02-12 15:00:22,645 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676214007285-1676214022591 2023-02-12 15:00:22,645 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676214007285-1676214022591 2023-02-12 15:00:22,663 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676214007285-1676214022591 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/hbase-failsafe-snaptb1-1676214007285-1676214022591 2023-02-12 15:00:22,678 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676214007285-1676214022591 of table testRestoreSnapshot-1676214007285 completed 2023-02-12 15:00:22,678 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676214007285-1676214022591 2023-02-12 15:00:22,679 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676214007285-1676214022591 2023-02-12 15:00:22,679 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676214007285-1676214022591 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676214007285 at 1676214022605Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676214007285-1676214022591 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } at 1676214022614 (+9 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676214007285-1676214022591 at 1676214022645 (+31 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676214007285-1676214022591 at 1676214022645Writing data manifest for hbase-failsafe-snaptb1-1676214007285-1676214022591 at 1676214022649 (+4 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676214007285-1676214022591 at 1676214022657 (+8 ms)Snapshot hbase-failsafe-snaptb1-1676214007285-1676214022591 of table testRestoreSnapshot-1676214007285 completed at 1676214022678 (+21 ms) 2023-02-12 15:00:22,680 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE 2023-02-12 15:00:22,682 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676214007285, type=EXCLUSIVE in 77 msec 2023-02-12 15:00:22,856 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 15:00:22,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676214007285-1676214022591 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 } is done 2023-02-12 15:00:22,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676214007285-1676214022591 table=testRestoreSnapshot-1676214007285 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 15:00:22,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676214007285 snapshot=name: "snaptb1-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214016742 type: DISABLED version: 2 ttl: 0 ) 2023-02-12 15:00:22,872 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676214007285 as table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-12 15:00:22,884 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/.tabledesc/.tableinfo.0000000004 2023-02-12 15:00:22,885 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/.tabledesc/.tableinfo.0000000003 2023-02-12 15:00:22,885 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/.tabledesc/.tableinfo.0000000004 2023-02-12 15:00:22,885 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } 2023-02-12 15:00:22,888 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214016742 type: DISABLED version: 2 ttl: 0 2023-02-12 15:00:22,888 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,895 DEBUG [PEWorker-2] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,896 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 of snapshot=snaptb1-1676214007285 to region=e793c55fa93954c06e9823103f4574b2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,899 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 of snapshot=snaptb1-1676214007285 to region=e793c55fa93954c06e9823103f4574b2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,902 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 of snapshot=snaptb1-1676214007285 to region=e793c55fa93954c06e9823103f4574b2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,905 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 of snapshot=snaptb1-1676214007285 to region=e793c55fa93954c06e9823103f4574b2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,908 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c of snapshot=snaptb1-1676214007285 to region=e793c55fa93954c06e9823103f4574b2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,911 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 of snapshot=snaptb1-1676214007285 to region=e793c55fa93954c06e9823103f4574b2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,915 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 of snapshot=snaptb1-1676214007285 to region=e793c55fa93954c06e9823103f4574b2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,918 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b of snapshot=snaptb1-1676214007285 to region=e793c55fa93954c06e9823103f4574b2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,922 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a of snapshot=snaptb1-1676214007285 to region=e793c55fa93954c06e9823103f4574b2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,925 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 of snapshot=snaptb1-1676214007285 to region=e793c55fa93954c06e9823103f4574b2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,928 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 of snapshot=snaptb1-1676214007285 to region=e793c55fa93954c06e9823103f4574b2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,931 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f of snapshot=snaptb1-1676214007285 to region=e793c55fa93954c06e9823103f4574b2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,934 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:22,934 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:22,934 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:22,934 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:22,934 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:22,934 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(223): region to restore: e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:22,936 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 9837fd0f2c07404bbfa672c073f42881 of snapshot=snaptb1-1676214007285 to region=a2b878baabc37b3076d648a696d2d33f table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,936 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 49b9c7e6778343f982623f169324b842 of snapshot=snaptb1-1676214007285 to region=e116768b6cb1e8fb57dfa7c68914d39c table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,936 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 984b06764f54429d98a03a4b3659e10a of snapshot=snaptb1-1676214007285 to region=469de65a1b57113b3e7565d5f5bf8201 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,936 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 3a1b76d129894ec2adcd309280852eb4 of snapshot=snaptb1-1676214007285 to region=c70c08e0263941ae1bfbb2d57f1c3052 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,936 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 393173ee6d644dfbb763f3ba123cdb44 of snapshot=snaptb1-1676214007285 to region=87a15dc17c09c51160706b761e764ba2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,936 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 4a211d423a9d40e08d63ef96097f6015 of snapshot=snaptb1-1676214007285 to region=e31ecbd7855a2e0bca06cbe75ecfc8fe table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,942 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink b00b668a6b014eb4a4e4c4bb861069f4 of snapshot=snaptb1-1676214007285 to region=e116768b6cb1e8fb57dfa7c68914d39c table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,942 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 5718cdbb97f24b3081185dfde2fb1e55 of snapshot=snaptb1-1676214007285 to region=87a15dc17c09c51160706b761e764ba2 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,943 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink f5c092a8a2e84ffe91bb687f3ea1c42d of snapshot=snaptb1-1676214007285 to region=a2b878baabc37b3076d648a696d2d33f table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,943 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 498d9c0eb3304fc7aedc40e7676473e2 of snapshot=snaptb1-1676214007285 to region=c70c08e0263941ae1bfbb2d57f1c3052 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,943 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a67854a9941340869b744861955b1505 of snapshot=snaptb1-1676214007285 to region=e31ecbd7855a2e0bca06cbe75ecfc8fe table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,944 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e0edae35a0ff4d6db2a54417a26dc5a6 of snapshot=snaptb1-1676214007285 to region=469de65a1b57113b3e7565d5f5bf8201 table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:22,955 INFO [PEWorker-2] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214016742 type: DISABLED version: 2 ttl: 0 2023-02-12 15:00:22,956 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214022956"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:22,956 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214022956"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:22,956 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214022956"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:22,956 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214022956"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:22,957 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214022956"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:22,957 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214022956"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:22,959 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-12 15:00:22,959 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-12 15:00:22,959 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214022957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214022957"}]},"ts":"1676214022957"} 2023-02-12 15:00:22,959 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214022957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214022957"}]},"ts":"1676214022957"} 2023-02-12 15:00:22,959 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214022957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214022957"}]},"ts":"1676214022957"} 2023-02-12 15:00:22,959 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214022957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214022957"}]},"ts":"1676214022957"} 2023-02-12 15:00:22,959 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214022957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214022957"}]},"ts":"1676214022957"} 2023-02-12 15:00:22,959 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214022957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214022957"}]},"ts":"1676214022957"} 2023-02-12 15:00:22,961 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-12 15:00:22,961 INFO [PEWorker-2] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-12 15:00:22,961 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-12 15:00:22,961 INFO [PEWorker-2] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676214007285 table=testRestoreSnapshot-1676214007285 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676214007285 completed! 2023-02-12 15:00:22,963 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676214007285 snapshot=name: "snaptb1-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214016742 type: DISABLED version: 2 ttl: 0 ) in 92 msec 2023-02-12 15:00:23,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-12 15:00:23,125 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676214007285, procId: 147 completed 2023-02-12 15:00:23,125 INFO [Listener at localhost/38323] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676214007285-1676214022591 2023-02-12 15:00:23,126 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb1-1676214007285-1676214022591" 2023-02-12 15:00:23,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676214007285-1676214022591 2023-02-12 15:00:23,129 INFO [Listener at localhost/38323] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676214007285 2023-02-12 15:00:23,129 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676214007285 2023-02-12 15:00:23,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:23,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-12 15:00:23,133 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676214007285 2023-02-12 15:00:23,133 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214023133"}]},"ts":"1676214023133"} 2023-02-12 15:00:23,134 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLING in hbase:meta 2023-02-12 15:00:23,136 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 15:00:23,136 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN}] 2023-02-12 15:00:23,138 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN 2023-02-12 15:00:23,138 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN 2023-02-12 15:00:23,138 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN 2023-02-12 15:00:23,138 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN 2023-02-12 15:00:23,138 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN 2023-02-12 15:00:23,139 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:23,139 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:23,139 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:23,159 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:23,159 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:23,160 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN 2023-02-12 15:00:23,161 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-12 15:00:23,161 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6452cab0 to 127.0.0.1:61893 2023-02-12 15:00:23,161 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:23,162 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0aebe9d0 to 127.0.0.1:61893 2023-02-12 15:00:23,162 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:23,289 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-12 15:00:23,289 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 15:00:23,289 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 15:00:23,289 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 15:00:23,289 DEBUG [jenkins-hbase4:45369] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 15:00:23,292 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:23,292 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:23,293 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214023292"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214023292"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214023292"}]},"ts":"1676214023292"} 2023-02-12 15:00:23,293 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214023292"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214023292"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214023292"}]},"ts":"1676214023292"} 2023-02-12 15:00:23,292 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:23,292 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:23,293 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214023292"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214023292"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214023292"}]},"ts":"1676214023292"} 2023-02-12 15:00:23,293 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214023292"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214023292"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214023292"}]},"ts":"1676214023292"} 2023-02-12 15:00:23,292 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:23,293 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214023292"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214023292"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214023292"}]},"ts":"1676214023292"} 2023-02-12 15:00:23,294 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=149, state=RUNNABLE; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:23,295 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=153, state=RUNNABLE; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:23,295 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=150, state=RUNNABLE; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:23,296 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=154, state=RUNNABLE; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:23,297 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=152, state=RUNNABLE; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:23,297 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:23,298 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214023297"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214023297"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214023297"}]},"ts":"1676214023297"} 2023-02-12 15:00:23,299 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=151, state=RUNNABLE; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:23,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-12 15:00:23,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:23,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 15:00:23,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:23,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:23,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:23,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:23,449 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:23,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 15:00:23,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:23,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:23,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:23,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:23,450 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:23,451 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:23,451 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:23,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 15:00:23,451 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:23,451 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:23,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:23,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:23,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:23,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:23,451 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 469de65a1b57113b3e7565d5f5bf8201 columnFamilyName cf 2023-02-12 15:00:23,452 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:23,452 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:23,452 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:23,453 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e31ecbd7855a2e0bca06cbe75ecfc8fe columnFamilyName cf 2023-02-12 15:00:23,453 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:23,453 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:23,453 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e116768b6cb1e8fb57dfa7c68914d39c columnFamilyName cf 2023-02-12 15:00:23,469 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-4a211d423a9d40e08d63ef96097f6015 2023-02-12 15:00:23,469 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-984b06764f54429d98a03a4b3659e10a 2023-02-12 15:00:23,469 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-49b9c7e6778343f982623f169324b842 2023-02-12 15:00:23,478 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-b00b668a6b014eb4a4e4c4bb861069f4 2023-02-12 15:00:23,478 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(310): Store=e116768b6cb1e8fb57dfa7c68914d39c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:23,479 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-e0edae35a0ff4d6db2a54417a26dc5a6 2023-02-12 15:00:23,479 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(310): Store=469de65a1b57113b3e7565d5f5bf8201/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:23,479 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:23,480 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:23,481 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:23,482 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:23,482 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-a67854a9941340869b744861955b1505 2023-02-12 15:00:23,482 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(310): Store=e31ecbd7855a2e0bca06cbe75ecfc8fe/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:23,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:23,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:23,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:23,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:23,486 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 469de65a1b57113b3e7565d5f5bf8201; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71884411, jitterRate=0.0711611956357956}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:23,486 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e116768b6cb1e8fb57dfa7c68914d39c; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74680105, jitterRate=0.11282028257846832}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:23,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 469de65a1b57113b3e7565d5f5bf8201: Running coprocessor pre-open hook at 1676214023449Writing region info on filesystem at 1676214023449Initializing all the Stores at 1676214023450 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214023450Cleaning up temporary data from old regions at 1676214023482 (+32 ms)Cleaning up detritus from prior splits at 1676214023483 (+1 ms)Running coprocessor post-open hooks at 1676214023486 (+3 ms)Region opened successfully at 1676214023486 2023-02-12 15:00:23,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e116768b6cb1e8fb57dfa7c68914d39c: Running coprocessor pre-open hook at 1676214023451Writing region info on filesystem at 1676214023451Initializing all the Stores at 1676214023452 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214023452Cleaning up temporary data from old regions at 1676214023482 (+30 ms)Cleaning up detritus from prior splits at 1676214023482Running coprocessor post-open hooks at 1676214023486 (+4 ms)Region opened successfully at 1676214023486 2023-02-12 15:00:23,487 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201., pid=155, masterSystemTime=1676214023446 2023-02-12 15:00:23,487 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c., pid=158, masterSystemTime=1676214023448 2023-02-12 15:00:23,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:23,488 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:23,488 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e31ecbd7855a2e0bca06cbe75ecfc8fe; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72414867, jitterRate=0.07906560599803925}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:23,489 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:23,489 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:23,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Running coprocessor pre-open hook at 1676214023450Writing region info on filesystem at 1676214023450Initializing all the Stores at 1676214023450Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214023450Cleaning up temporary data from old regions at 1676214023485 (+35 ms)Cleaning up detritus from prior splits at 1676214023485Running coprocessor post-open hooks at 1676214023489 (+4 ms)Region opened successfully at 1676214023489 2023-02-12 15:00:23,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 15:00:23,489 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:23,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:23,489 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214023489"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214023489"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214023489"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214023489"}]},"ts":"1676214023489"} 2023-02-12 15:00:23,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:23,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:23,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:23,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:23,490 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:23,490 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:23,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 15:00:23,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:23,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:23,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:23,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:23,491 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe., pid=157, masterSystemTime=1676214023447 2023-02-12 15:00:23,491 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:23,491 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214023491"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214023491"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214023491"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214023491"}]},"ts":"1676214023491"} 2023-02-12 15:00:23,492 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:23,492 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:23,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:23,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:23,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:23,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 15:00:23,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:23,494 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:23,494 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:23,494 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:23,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:23,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:23,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:23,494 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:23,494 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c70c08e0263941ae1bfbb2d57f1c3052 columnFamilyName cf 2023-02-12 15:00:23,495 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 87a15dc17c09c51160706b761e764ba2 columnFamilyName cf 2023-02-12 15:00:23,495 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:23,496 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:23,496 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214023495"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214023495"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214023495"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214023495"}]},"ts":"1676214023495"} 2023-02-12 15:00:23,498 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=149 2023-02-12 15:00:23,498 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=149, state=SUCCESS; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,37113,1676214004377 in 200 msec 2023-02-12 15:00:23,499 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=154 2023-02-12 15:00:23,499 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=154, state=SUCCESS; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423 in 200 msec 2023-02-12 15:00:23,500 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN in 362 msec 2023-02-12 15:00:23,501 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:23,501 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:23,501 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN in 363 msec 2023-02-12 15:00:23,501 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=150 2023-02-12 15:00:23,501 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=150, state=SUCCESS; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,38453,1676214004461 in 204 msec 2023-02-12 15:00:23,502 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a2b878baabc37b3076d648a696d2d33f columnFamilyName cf 2023-02-12 15:00:23,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN in 365 msec 2023-02-12 15:00:23,513 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-393173ee6d644dfbb763f3ba123cdb44 2023-02-12 15:00:23,516 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-3a1b76d129894ec2adcd309280852eb4 2023-02-12 15:00:23,516 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-9837fd0f2c07404bbfa672c073f42881 2023-02-12 15:00:23,528 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-5718cdbb97f24b3081185dfde2fb1e55 2023-02-12 15:00:23,528 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(310): Store=87a15dc17c09c51160706b761e764ba2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:23,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:23,529 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-498d9c0eb3304fc7aedc40e7676473e2 2023-02-12 15:00:23,530 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(310): Store=c70c08e0263941ae1bfbb2d57f1c3052/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:23,530 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-f5c092a8a2e84ffe91bb687f3ea1c42d 2023-02-12 15:00:23,530 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(310): Store=a2b878baabc37b3076d648a696d2d33f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:23,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:23,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:23,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:23,532 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:23,532 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:23,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:23,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:23,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:23,535 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 87a15dc17c09c51160706b761e764ba2; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69214083, jitterRate=0.03137020766735077}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:23,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 87a15dc17c09c51160706b761e764ba2: Running coprocessor pre-open hook at 1676214023490Writing region info on filesystem at 1676214023490Initializing all the Stores at 1676214023491 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214023491Cleaning up temporary data from old regions at 1676214023531 (+40 ms)Cleaning up detritus from prior splits at 1676214023532 (+1 ms)Running coprocessor post-open hooks at 1676214023535 (+3 ms)Region opened successfully at 1676214023535 2023-02-12 15:00:23,535 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a2b878baabc37b3076d648a696d2d33f; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63286545, jitterRate=-0.05695699155330658}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:23,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a2b878baabc37b3076d648a696d2d33f: Running coprocessor pre-open hook at 1676214023494Writing region info on filesystem at 1676214023494Initializing all the Stores at 1676214023495 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214023495Cleaning up temporary data from old regions at 1676214023532 (+37 ms)Cleaning up detritus from prior splits at 1676214023533 (+1 ms)Running coprocessor post-open hooks at 1676214023535 (+2 ms)Region opened successfully at 1676214023535 2023-02-12 15:00:23,536 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2., pid=159, masterSystemTime=1676214023446 2023-02-12 15:00:23,536 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c70c08e0263941ae1bfbb2d57f1c3052; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73909598, jitterRate=0.10133883357048035}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:23,536 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f., pid=156, masterSystemTime=1676214023447 2023-02-12 15:00:23,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c70c08e0263941ae1bfbb2d57f1c3052: Running coprocessor pre-open hook at 1676214023490Writing region info on filesystem at 1676214023490Initializing all the Stores at 1676214023491 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214023491Cleaning up temporary data from old regions at 1676214023532 (+41 ms)Cleaning up detritus from prior splits at 1676214023533 (+1 ms)Running coprocessor post-open hooks at 1676214023536 (+3 ms)Region opened successfully at 1676214023536 2023-02-12 15:00:23,537 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052., pid=160, masterSystemTime=1676214023448 2023-02-12 15:00:23,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:23,538 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:23,538 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:23,539 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214023538"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214023538"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214023538"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214023538"}]},"ts":"1676214023538"} 2023-02-12 15:00:23,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:23,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:23,539 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:23,539 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214023539"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214023539"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214023539"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214023539"}]},"ts":"1676214023539"} 2023-02-12 15:00:23,540 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:23,540 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214023540"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214023540"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214023540"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214023540"}]},"ts":"1676214023540"} 2023-02-12 15:00:23,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:23,541 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:23,543 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=152 2023-02-12 15:00:23,543 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=152, state=SUCCESS; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,37113,1676214004377 in 244 msec 2023-02-12 15:00:23,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=153 2023-02-12 15:00:23,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=153, state=SUCCESS; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,38453,1676214004461 in 247 msec 2023-02-12 15:00:23,545 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN in 407 msec 2023-02-12 15:00:23,545 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=151 2023-02-12 15:00:23,545 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=151, state=SUCCESS; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,33845,1676214004423 in 244 msec 2023-02-12 15:00:23,545 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN in 408 msec 2023-02-12 15:00:23,546 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=151, resume processing ppid=148 2023-02-12 15:00:23,546 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN in 409 msec 2023-02-12 15:00:23,547 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214023547"}]},"ts":"1676214023547"} 2023-02-12 15:00:23,548 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLED in hbase:meta 2023-02-12 15:00:23,550 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676214007285' was successfully enabled. 2023-02-12 15:00:23,551 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676214007285 in 421 msec 2023-02-12 15:00:23,803 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676214007285' 2023-02-12 15:00:23,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-12 15:00:23,884 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 148 completed 2023-02-12 15:00:23,885 DEBUG [Listener at localhost/38323] zookeeper.ReadOnlyZKClient(139): Connect 0x1550b85e to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:23,890 DEBUG [Listener at localhost/38323] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ed0e7fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:23,891 DEBUG [hconnection-0x70b95fe7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:23,894 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53050, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:23,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:23,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,134 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 15:00:24,137 DEBUG [hconnection-0x70b95fe7-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:24,141 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37148, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:24,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,362 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 15:00:24,365 DEBUG [hconnection-0x70b95fe7-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:24,369 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53474, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:24,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,554 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 15:00:24,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,776 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 15:00:24,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,975 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 15:00:24,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:24,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:25,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:26,892 INFO [Listener at localhost/38323] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676214007285 2023-02-12 15:00:26,893 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676214007285 2023-02-12 15:00:26,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:26,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-12 15:00:26,897 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214026896"}]},"ts":"1676214026896"} 2023-02-12 15:00:26,898 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLING in hbase:meta 2023-02-12 15:00:26,900 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676214007285 to state=DISABLING 2023-02-12 15:00:26,901 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN}] 2023-02-12 15:00:26,903 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN 2023-02-12 15:00:26,903 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN 2023-02-12 15:00:26,903 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN 2023-02-12 15:00:26,903 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN 2023-02-12 15:00:26,903 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN 2023-02-12 15:00:26,903 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:26,903 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:26,903 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214026903"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214026903"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214026903"}]},"ts":"1676214026903"} 2023-02-12 15:00:26,904 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214026903"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214026903"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214026903"}]},"ts":"1676214026903"} 2023-02-12 15:00:26,903 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:26,903 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:26,904 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214026903"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214026903"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214026903"}]},"ts":"1676214026903"} 2023-02-12 15:00:26,904 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214026903"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214026903"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214026903"}]},"ts":"1676214026903"} 2023-02-12 15:00:26,904 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:26,904 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214026904"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214026904"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214026904"}]},"ts":"1676214026904"} 2023-02-12 15:00:26,905 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=162, state=RUNNABLE; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:26,905 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=164, state=RUNNABLE; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:26,906 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=165, state=RUNNABLE; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:26,907 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=163, state=RUNNABLE; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:26,908 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=167, state=RUNNABLE; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:26,909 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN 2023-02-12 15:00:26,910 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:26,910 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214026909"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214026909"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214026909"}]},"ts":"1676214026909"} 2023-02-12 15:00:26,911 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=166, state=RUNNABLE; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:27,057 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:27,057 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:27,059 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e116768b6cb1e8fb57dfa7c68914d39c, disabling compactions & flushes 2023-02-12 15:00:27,059 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 469de65a1b57113b3e7565d5f5bf8201, disabling compactions & flushes 2023-02-12 15:00:27,059 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:27,059 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:27,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:27,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:27,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. after waiting 0 ms 2023-02-12 15:00:27,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. after waiting 0 ms 2023-02-12 15:00:27,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:27,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:27,060 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:27,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e31ecbd7855a2e0bca06cbe75ecfc8fe, disabling compactions & flushes 2023-02-12 15:00:27,061 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:27,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:27,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. after waiting 0 ms 2023-02-12 15:00:27,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:27,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-12 15:00:27,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-12 15:00:27,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-12 15:00:27,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:27,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Waiting for close lock at 1676214027060Running coprocessor pre-close hooks at 1676214027060Disabling compacts and flushes for region at 1676214027060Disabling writes for close at 1676214027061 (+1 ms)Writing region close event to WAL at 1676214027066 (+5 ms)Running coprocessor post-close hooks at 1676214027069 (+3 ms)Closed at 1676214027069 2023-02-12 15:00:27,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:27,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:27,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 469de65a1b57113b3e7565d5f5bf8201: Waiting for close lock at 1676214027059Running coprocessor pre-close hooks at 1676214027059Disabling compacts and flushes for region at 1676214027059Disabling writes for close at 1676214027060 (+1 ms)Writing region close event to WAL at 1676214027066 (+6 ms)Running coprocessor post-close hooks at 1676214027069 (+3 ms)Closed at 1676214027069 2023-02-12 15:00:27,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e116768b6cb1e8fb57dfa7c68914d39c: Waiting for close lock at 1676214027058Running coprocessor pre-close hooks at 1676214027058Disabling compacts and flushes for region at 1676214027058Disabling writes for close at 1676214027060 (+2 ms)Writing region close event to WAL at 1676214027065 (+5 ms)Running coprocessor post-close hooks at 1676214027069 (+4 ms)Closed at 1676214027069 2023-02-12 15:00:27,070 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:27,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:27,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a2b878baabc37b3076d648a696d2d33f, disabling compactions & flushes 2023-02-12 15:00:27,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:27,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:27,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. after waiting 0 ms 2023-02-12 15:00:27,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:27,071 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSED 2023-02-12 15:00:27,071 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027071"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214027071"}]},"ts":"1676214027071"} 2023-02-12 15:00:27,074 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:27,075 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:27,076 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c70c08e0263941ae1bfbb2d57f1c3052, disabling compactions & flushes 2023-02-12 15:00:27,076 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSED 2023-02-12 15:00:27,076 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:27,076 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214027076"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214027076"}]},"ts":"1676214027076"} 2023-02-12 15:00:27,076 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:27,076 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:27,076 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:27,076 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. after waiting 0 ms 2023-02-12 15:00:27,077 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:27,077 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 87a15dc17c09c51160706b761e764ba2, disabling compactions & flushes 2023-02-12 15:00:27,077 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:27,077 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:27,077 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. after waiting 0 ms 2023-02-12 15:00:27,077 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:27,077 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSED 2023-02-12 15:00:27,077 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214027077"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214027077"}]},"ts":"1676214027077"} 2023-02-12 15:00:27,085 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-12 15:00:27,085 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=163 2023-02-12 15:00:27,085 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=163, state=SUCCESS; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,38453,1676214004461 in 170 msec 2023-02-12 15:00:27,086 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:27,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a2b878baabc37b3076d648a696d2d33f: Waiting for close lock at 1676214027071Running coprocessor pre-close hooks at 1676214027071Disabling compacts and flushes for region at 1676214027071Disabling writes for close at 1676214027071Writing region close event to WAL at 1676214027075 (+4 ms)Running coprocessor post-close hooks at 1676214027086 (+11 ms)Closed at 1676214027086 2023-02-12 15:00:27,086 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=167 2023-02-12 15:00:27,086 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=167, state=SUCCESS; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,33845,1676214004423 in 175 msec 2023-02-12 15:00:27,087 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN in 184 msec 2023-02-12 15:00:27,087 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:27,087 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-12 15:00:27,087 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=162 2023-02-12 15:00:27,087 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN in 185 msec 2023-02-12 15:00:27,087 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=162, state=SUCCESS; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,37113,1676214004377 in 180 msec 2023-02-12 15:00:27,087 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSED 2023-02-12 15:00:27,088 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027087"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214027087"}]},"ts":"1676214027087"} 2023-02-12 15:00:27,088 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:27,088 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 87a15dc17c09c51160706b761e764ba2: Waiting for close lock at 1676214027076Running coprocessor pre-close hooks at 1676214027076Disabling compacts and flushes for region at 1676214027076Disabling writes for close at 1676214027077 (+1 ms)Writing region close event to WAL at 1676214027084 (+7 ms)Running coprocessor post-close hooks at 1676214027088 (+4 ms)Closed at 1676214027088 2023-02-12 15:00:27,088 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN in 186 msec 2023-02-12 15:00:27,089 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:27,089 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSED 2023-02-12 15:00:27,090 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027089"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214027089"}]},"ts":"1676214027089"} 2023-02-12 15:00:27,095 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-12 15:00:27,095 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:27,096 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c70c08e0263941ae1bfbb2d57f1c3052: Waiting for close lock at 1676214027075Running coprocessor pre-close hooks at 1676214027075Disabling compacts and flushes for region at 1676214027075Disabling writes for close at 1676214027077 (+2 ms)Writing region close event to WAL at 1676214027082 (+5 ms)Running coprocessor post-close hooks at 1676214027095 (+13 ms)Closed at 1676214027095 2023-02-12 15:00:27,096 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=166 2023-02-12 15:00:27,096 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=166, state=SUCCESS; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,38453,1676214004461 in 178 msec 2023-02-12 15:00:27,097 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=165 2023-02-12 15:00:27,097 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=165, state=SUCCESS; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,37113,1676214004377 in 185 msec 2023-02-12 15:00:27,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:27,097 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN in 195 msec 2023-02-12 15:00:27,097 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSED 2023-02-12 15:00:27,097 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027097"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214027097"}]},"ts":"1676214027097"} 2023-02-12 15:00:27,098 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN in 196 msec 2023-02-12 15:00:27,100 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=164 2023-02-12 15:00:27,100 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=164, state=SUCCESS; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,33845,1676214004423 in 194 msec 2023-02-12 15:00:27,101 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=164, resume processing ppid=161 2023-02-12 15:00:27,101 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN in 199 msec 2023-02-12 15:00:27,102 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214027102"}]},"ts":"1676214027102"} 2023-02-12 15:00:27,103 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLED in hbase:meta 2023-02-12 15:00:27,105 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676214007285 to state=DISABLED 2023-02-12 15:00:27,106 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676214007285 in 212 msec 2023-02-12 15:00:27,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-12 15:00:27,148 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 161 completed 2023-02-12 15:00:27,152 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,159 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,161 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-12 15:00:27,167 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:27,167 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:27,167 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:27,167 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:27,167 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:27,167 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:27,169 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits] 2023-02-12 15:00:27,169 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits] 2023-02-12 15:00:27,169 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits] 2023-02-12 15:00:27,169 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits] 2023-02-12 15:00:27,170 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits] 2023-02-12 15:00:27,170 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits] 2023-02-12 15:00:27,173 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-49b9c7e6778343f982623f169324b842 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-49b9c7e6778343f982623f169324b842 2023-02-12 15:00:27,173 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-4a211d423a9d40e08d63ef96097f6015 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-4a211d423a9d40e08d63ef96097f6015 2023-02-12 15:00:27,173 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-9837fd0f2c07404bbfa672c073f42881 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-9837fd0f2c07404bbfa672c073f42881 2023-02-12 15:00:27,174 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-3a1b76d129894ec2adcd309280852eb4 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-3a1b76d129894ec2adcd309280852eb4 2023-02-12 15:00:27,174 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-393173ee6d644dfbb763f3ba123cdb44 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-393173ee6d644dfbb763f3ba123cdb44 2023-02-12 15:00:27,174 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-984b06764f54429d98a03a4b3659e10a to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-984b06764f54429d98a03a4b3659e10a 2023-02-12 15:00:27,175 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-5718cdbb97f24b3081185dfde2fb1e55 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-5718cdbb97f24b3081185dfde2fb1e55 2023-02-12 15:00:27,175 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-a67854a9941340869b744861955b1505 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-a67854a9941340869b744861955b1505 2023-02-12 15:00:27,176 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-b00b668a6b014eb4a4e4c4bb861069f4 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-b00b668a6b014eb4a4e4c4bb861069f4 2023-02-12 15:00:27,176 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-498d9c0eb3304fc7aedc40e7676473e2 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-498d9c0eb3304fc7aedc40e7676473e2 2023-02-12 15:00:27,176 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-f5c092a8a2e84ffe91bb687f3ea1c42d to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-f5c092a8a2e84ffe91bb687f3ea1c42d 2023-02-12 15:00:27,176 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-e0edae35a0ff4d6db2a54417a26dc5a6 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-e0edae35a0ff4d6db2a54417a26dc5a6 2023-02-12 15:00:27,179 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits/27.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits/27.seqid 2023-02-12 15:00:27,179 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits/27.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits/27.seqid 2023-02-12 15:00:27,180 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits/27.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits/27.seqid 2023-02-12 15:00:27,180 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits/27.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits/27.seqid 2023-02-12 15:00:27,180 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits/27.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits/27.seqid 2023-02-12 15:00:27,180 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:27,181 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:27,181 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:27,181 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:27,181 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:27,181 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits/27.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits/27.seqid 2023-02-12 15:00:27,181 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:27,181 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676214007285 regions 2023-02-12 15:00:27,182 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2 2023-02-12 15:00:27,182 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf] 2023-02-12 15:00:27,184 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 2023-02-12 15:00:27,185 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 2023-02-12 15:00:27,186 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 2023-02-12 15:00:27,187 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 2023-02-12 15:00:27,188 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c 2023-02-12 15:00:27,189 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 2023-02-12 15:00:27,190 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 2023-02-12 15:00:27,192 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b 2023-02-12 15:00:27,193 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a 2023-02-12 15:00:27,194 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 2023-02-12 15:00:27,195 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 2023-02-12 15:00:27,196 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f 2023-02-12 15:00:27,196 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2 2023-02-12 15:00:27,198 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,221 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676214007285 from hbase:meta 2023-02-12 15:00:27,223 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676214007285' descriptor. 2023-02-12 15:00:27,224 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,224 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676214007285' from region states. 2023-02-12 15:00:27,225 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214027224"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:27,225 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214027224"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:27,225 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214027224"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:27,225 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214027224"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:27,225 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214027224"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:27,225 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214027224"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:27,227 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-12 15:00:27,227 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''}] 2023-02-12 15:00:27,227 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676214007285' as deleted. 2023-02-12 15:00:27,227 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676214027227"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:27,228 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676214007285 state from META 2023-02-12 15:00:27,230 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,231 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 in 77 msec 2023-02-12 15:00:27,353 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-12 15:00:27,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-12 15:00:27,413 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 174 completed 2023-02-12 15:00:27,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676214007285 snapshot=name: "snaptb1-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214016742 type: DISABLED version: 2 ttl: 0 ) 2023-02-12 15:00:27,443 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676214007285 as table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-12 15:00:27,456 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/.tabledesc/.tableinfo.0000000001 2023-02-12 15:00:27,457 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214016742 type: DISABLED version: 2 ttl: 0 2023-02-12 15:00:27,457 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,458 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,460 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,462 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,464 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,466 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,467 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,469 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,471 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,472 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,474 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,476 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,477 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(239): region to add: e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:27,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(239): region to add: c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:27,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(239): region to add: a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:27,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(239): region to add: e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:27,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(239): region to add: 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:27,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(239): region to add: 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:27,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(585): clone region=e116768b6cb1e8fb57dfa7c68914d39c as e116768b6cb1e8fb57dfa7c68914d39c in snapshot snaptb1-1676214007285 2023-02-12 15:00:27,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(585): clone region=c70c08e0263941ae1bfbb2d57f1c3052 as c70c08e0263941ae1bfbb2d57f1c3052 in snapshot snaptb1-1676214007285 2023-02-12 15:00:27,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(585): clone region=a2b878baabc37b3076d648a696d2d33f as a2b878baabc37b3076d648a696d2d33f in snapshot snaptb1-1676214007285 2023-02-12 15:00:27,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(585): clone region=e31ecbd7855a2e0bca06cbe75ecfc8fe as e31ecbd7855a2e0bca06cbe75ecfc8fe in snapshot snaptb1-1676214007285 2023-02-12 15:00:27,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(585): clone region=469de65a1b57113b3e7565d5f5bf8201 as 469de65a1b57113b3e7565d5f5bf8201 in snapshot snaptb1-1676214007285 2023-02-12 15:00:27,479 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(585): clone region=87a15dc17c09c51160706b761e764ba2 as 87a15dc17c09c51160706b761e764ba2 in snapshot snaptb1-1676214007285 2023-02-12 15:00:27,481 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676214007285', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:27,481 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676214007285', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:27,482 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676214007285', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:27,482 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676214007285', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:27,482 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676214007285', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:27,482 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676214007285', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:27,508 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:27,508 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 3a1b76d129894ec2adcd309280852eb4 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,520 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:27,521 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 4a211d423a9d40e08d63ef96097f6015 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,524 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:27,524 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 9837fd0f2c07404bbfa672c073f42881 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,532 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:27,532 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 49b9c7e6778343f982623f169324b842 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,532 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a67854a9941340869b744861955b1505 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,533 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 498d9c0eb3304fc7aedc40e7676473e2 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,534 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink f5c092a8a2e84ffe91bb687f3ea1c42d from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,536 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:27,536 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 393173ee6d644dfbb763f3ba123cdb44 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,537 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink b00b668a6b014eb4a4e4c4bb861069f4 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,537 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing e31ecbd7855a2e0bca06cbe75ecfc8fe, disabling compactions & flushes 2023-02-12 15:00:27,537 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:27,537 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:27,537 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. after waiting 0 ms 2023-02-12 15:00:27,537 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:27,537 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:27,537 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Waiting for close lock at 1676214027537Disabling compacts and flushes for region at 1676214027537Disabling writes for close at 1676214027537Writing region close event to WAL at 1676214027537Closed at 1676214027537 2023-02-12 15:00:27,540 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:27,540 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 984b06764f54429d98a03a4b3659e10a from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,541 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing c70c08e0263941ae1bfbb2d57f1c3052, disabling compactions & flushes 2023-02-12 15:00:27,541 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:27,541 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing a2b878baabc37b3076d648a696d2d33f, disabling compactions & flushes 2023-02-12 15:00:27,541 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:27,541 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:27,541 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. after waiting 0 ms 2023-02-12 15:00:27,541 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:27,541 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:27,541 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for a2b878baabc37b3076d648a696d2d33f: Waiting for close lock at 1676214027541Disabling compacts and flushes for region at 1676214027541Disabling writes for close at 1676214027541Writing region close event to WAL at 1676214027541Closed at 1676214027541 2023-02-12 15:00:27,541 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:27,541 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 5718cdbb97f24b3081185dfde2fb1e55 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,541 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing e116768b6cb1e8fb57dfa7c68914d39c, disabling compactions & flushes 2023-02-12 15:00:27,542 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:27,542 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:27,542 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. after waiting 0 ms 2023-02-12 15:00:27,542 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:27,542 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:27,542 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. after waiting 0 ms 2023-02-12 15:00:27,542 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:27,542 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for c70c08e0263941ae1bfbb2d57f1c3052: Waiting for close lock at 1676214027541Disabling compacts and flushes for region at 1676214027541Disabling writes for close at 1676214027542 (+1 ms)Writing region close event to WAL at 1676214027542Closed at 1676214027542 2023-02-12 15:00:27,542 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:27,542 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for e116768b6cb1e8fb57dfa7c68914d39c: Waiting for close lock at 1676214027541Disabling compacts and flushes for region at 1676214027541Disabling writes for close at 1676214027542 (+1 ms)Writing region close event to WAL at 1676214027542Closed at 1676214027542 2023-02-12 15:00:27,544 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e0edae35a0ff4d6db2a54417a26dc5a6 from cloned region in snapshot snaptb1-1676214007285 to table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:27,544 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing 87a15dc17c09c51160706b761e764ba2, disabling compactions & flushes 2023-02-12 15:00:27,544 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:27,544 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:27,544 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. after waiting 0 ms 2023-02-12 15:00:27,544 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:27,544 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:27,544 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for 87a15dc17c09c51160706b761e764ba2: Waiting for close lock at 1676214027544Disabling compacts and flushes for region at 1676214027544Disabling writes for close at 1676214027544Writing region close event to WAL at 1676214027544Closed at 1676214027544 2023-02-12 15:00:27,546 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing 469de65a1b57113b3e7565d5f5bf8201, disabling compactions & flushes 2023-02-12 15:00:27,546 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:27,546 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:27,546 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. after waiting 0 ms 2023-02-12 15:00:27,546 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:27,546 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:27,546 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for 469de65a1b57113b3e7565d5f5bf8201: Waiting for close lock at 1676214027546Disabling compacts and flushes for region at 1676214027546Disabling writes for close at 1676214027546Writing region close event to WAL at 1676214027546Closed at 1676214027546 2023-02-12 15:00:27,546 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214016742 type: DISABLED version: 2 ttl: 0 2023-02-12 15:00:27,546 INFO [PEWorker-1] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676214007285 on table=testRestoreSnapshot-1676214007285 completed! 2023-02-12 15:00:27,550 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214027550"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214027550"}]},"ts":"1676214027550"} 2023-02-12 15:00:27,550 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027550"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214027550"}]},"ts":"1676214027550"} 2023-02-12 15:00:27,550 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027550"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214027550"}]},"ts":"1676214027550"} 2023-02-12 15:00:27,550 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027550"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214027550"}]},"ts":"1676214027550"} 2023-02-12 15:00:27,551 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214027550"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214027550"}]},"ts":"1676214027550"} 2023-02-12 15:00:27,551 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027550"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214027550"}]},"ts":"1676214027550"} 2023-02-12 15:00:27,553 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-12 15:00:27,553 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214027553"}]},"ts":"1676214027553"} 2023-02-12 15:00:27,554 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLING in hbase:meta 2023-02-12 15:00:27,559 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-12 15:00:27,560 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 15:00:27,560 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 15:00:27,560 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 15:00:27,560 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 15:00:27,560 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN}] 2023-02-12 15:00:27,562 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN 2023-02-12 15:00:27,562 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN 2023-02-12 15:00:27,562 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN 2023-02-12 15:00:27,562 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN 2023-02-12 15:00:27,563 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN 2023-02-12 15:00:27,563 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=false 2023-02-12 15:00:27,563 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=false 2023-02-12 15:00:27,563 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=false 2023-02-12 15:00:27,563 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=false 2023-02-12 15:00:27,564 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=false 2023-02-12 15:00:27,564 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN 2023-02-12 15:00:27,565 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=false 2023-02-12 15:00:27,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-12 15:00:27,714 INFO [jenkins-hbase4:45369] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 15:00:27,717 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:27,717 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:27,717 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:27,717 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:27,717 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:27,717 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214027717"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214027717"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214027717"}]},"ts":"1676214027717"} 2023-02-12 15:00:27,717 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027717"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214027717"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214027717"}]},"ts":"1676214027717"} 2023-02-12 15:00:27,717 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027717"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214027717"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214027717"}]},"ts":"1676214027717"} 2023-02-12 15:00:27,717 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027717"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214027717"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214027717"}]},"ts":"1676214027717"} 2023-02-12 15:00:27,717 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214027717"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214027717"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214027717"}]},"ts":"1676214027717"} 2023-02-12 15:00:27,719 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=181, state=RUNNABLE; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:27,721 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=178, state=RUNNABLE; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:27,722 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=177, state=RUNNABLE; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:27,722 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=179, state=RUNNABLE; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:27,723 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=176, state=RUNNABLE; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:27,723 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:27,724 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027723"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214027723"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214027723"}]},"ts":"1676214027723"} 2023-02-12 15:00:27,725 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=180, state=RUNNABLE; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:27,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:27,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 15:00:27,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:27,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:27,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:27,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:27,876 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:27,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 15:00:27,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:27,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:27,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:27,876 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:27,876 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:27,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:27,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 15:00:27,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:27,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:27,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:27,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:27,877 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:27,878 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf 2023-02-12 15:00:27,878 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:27,878 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e116768b6cb1e8fb57dfa7c68914d39c columnFamilyName cf 2023-02-12 15:00:27,878 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:27,878 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:27,878 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf 2023-02-12 15:00:27,879 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:27,879 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c70c08e0263941ae1bfbb2d57f1c3052 columnFamilyName cf 2023-02-12 15:00:27,879 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf 2023-02-12 15:00:27,879 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 87a15dc17c09c51160706b761e764ba2 columnFamilyName cf 2023-02-12 15:00:27,889 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-49b9c7e6778343f982623f169324b842 2023-02-12 15:00:27,889 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-3a1b76d129894ec2adcd309280852eb4 2023-02-12 15:00:27,890 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-393173ee6d644dfbb763f3ba123cdb44 2023-02-12 15:00:27,895 DEBUG [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-b00b668a6b014eb4a4e4c4bb861069f4 2023-02-12 15:00:27,895 INFO [StoreOpener-e116768b6cb1e8fb57dfa7c68914d39c-1] regionserver.HStore(310): Store=e116768b6cb1e8fb57dfa7c68914d39c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:27,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:27,897 DEBUG [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-5718cdbb97f24b3081185dfde2fb1e55 2023-02-12 15:00:27,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:27,897 INFO [StoreOpener-87a15dc17c09c51160706b761e764ba2-1] regionserver.HStore(310): Store=87a15dc17c09c51160706b761e764ba2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:27,898 DEBUG [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-498d9c0eb3304fc7aedc40e7676473e2 2023-02-12 15:00:27,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:27,898 INFO [StoreOpener-c70c08e0263941ae1bfbb2d57f1c3052-1] regionserver.HStore(310): Store=c70c08e0263941ae1bfbb2d57f1c3052/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:27,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:27,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:27,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:27,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:27,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:27,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-12 15:00:27,902 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:27,902 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e116768b6cb1e8fb57dfa7c68914d39c; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61307923, jitterRate=-0.08644075691699982}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:27,902 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e116768b6cb1e8fb57dfa7c68914d39c: Running coprocessor pre-open hook at 1676214027876Writing region info on filesystem at 1676214027876Initializing all the Stores at 1676214027876Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214027876Cleaning up temporary data from old regions at 1676214027897 (+21 ms)Cleaning up detritus from prior splits at 1676214027898 (+1 ms)Running coprocessor post-open hooks at 1676214027902 (+4 ms)Region opened successfully at 1676214027902 2023-02-12 15:00:27,902 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-12 15:00:27,903 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 87a15dc17c09c51160706b761e764ba2; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74691324, jitterRate=0.1129874587059021}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:27,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 87a15dc17c09c51160706b761e764ba2: Running coprocessor pre-open hook at 1676214027877Writing region info on filesystem at 1676214027877Initializing all the Stores at 1676214027878 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214027878Cleaning up temporary data from old regions at 1676214027898 (+20 ms)Cleaning up detritus from prior splits at 1676214027899 (+1 ms)Running coprocessor post-open hooks at 1676214027903 (+4 ms)Region opened successfully at 1676214027903 2023-02-12 15:00:27,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c., pid=182, masterSystemTime=1676214027872 2023-02-12 15:00:27,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-12 15:00:27,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2., pid=185, masterSystemTime=1676214027874 2023-02-12 15:00:27,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c70c08e0263941ae1bfbb2d57f1c3052; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61201903, jitterRate=-0.08802057802677155}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:27,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c70c08e0263941ae1bfbb2d57f1c3052: Running coprocessor pre-open hook at 1676214027877Writing region info on filesystem at 1676214027877Initializing all the Stores at 1676214027877Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214027877Cleaning up temporary data from old regions at 1676214027899 (+22 ms)Cleaning up detritus from prior splits at 1676214027900 (+1 ms)Running coprocessor post-open hooks at 1676214027904 (+4 ms)Region opened successfully at 1676214027904 2023-02-12 15:00:27,905 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052., pid=183, masterSystemTime=1676214027873 2023-02-12 15:00:27,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:27,905 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:27,905 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:27,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 15:00:27,905 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:27,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:27,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:27,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:27,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:27,906 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214027905"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214027905"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214027905"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214027905"}]},"ts":"1676214027905"} 2023-02-12 15:00:27,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:27,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:27,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:27,906 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:27,906 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027906"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214027906"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214027906"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214027906"}]},"ts":"1676214027906"} 2023-02-12 15:00:27,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 15:00:27,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:27,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:27,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:27,907 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:27,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:27,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:27,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 15:00:27,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:27,907 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027906"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214027906"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214027906"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214027906"}]},"ts":"1676214027906"} 2023-02-12 15:00:27,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:27,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676214007285 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:27,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:27,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:27,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:27,908 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:27,908 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:27,908 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:27,909 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:27,909 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf 2023-02-12 15:00:27,909 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:27,909 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf 2023-02-12 15:00:27,910 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=181 2023-02-12 15:00:27,910 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=181, state=SUCCESS; OpenRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,37113,1676214004377 in 188 msec 2023-02-12 15:00:27,910 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 469de65a1b57113b3e7565d5f5bf8201 columnFamilyName cf 2023-02-12 15:00:27,910 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e31ecbd7855a2e0bca06cbe75ecfc8fe columnFamilyName cf 2023-02-12 15:00:27,911 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=179 2023-02-12 15:00:27,911 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=179, state=SUCCESS; OpenRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,33845,1676214004423 in 186 msec 2023-02-12 15:00:27,911 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=178 2023-02-12 15:00:27,911 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, ASSIGN in 350 msec 2023-02-12 15:00:27,911 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=178, state=SUCCESS; OpenRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461 in 190 msec 2023-02-12 15:00:27,912 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, ASSIGN in 351 msec 2023-02-12 15:00:27,913 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, ASSIGN in 351 msec 2023-02-12 15:00:27,917 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:27,917 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf 2023-02-12 15:00:27,918 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a2b878baabc37b3076d648a696d2d33f columnFamilyName cf 2023-02-12 15:00:27,924 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-984b06764f54429d98a03a4b3659e10a 2023-02-12 15:00:27,924 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-4a211d423a9d40e08d63ef96097f6015 2023-02-12 15:00:27,928 DEBUG [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-e0edae35a0ff4d6db2a54417a26dc5a6 2023-02-12 15:00:27,928 DEBUG [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-a67854a9941340869b744861955b1505 2023-02-12 15:00:27,932 INFO [StoreOpener-e31ecbd7855a2e0bca06cbe75ecfc8fe-1] regionserver.HStore(310): Store=e31ecbd7855a2e0bca06cbe75ecfc8fe/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:27,932 INFO [StoreOpener-469de65a1b57113b3e7565d5f5bf8201-1] regionserver.HStore(310): Store=469de65a1b57113b3e7565d5f5bf8201/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:27,932 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-9837fd0f2c07404bbfa672c073f42881 2023-02-12 15:00:27,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:27,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:27,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:27,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:27,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:27,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:27,937 DEBUG [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-f5c092a8a2e84ffe91bb687f3ea1c42d 2023-02-12 15:00:27,937 INFO [StoreOpener-a2b878baabc37b3076d648a696d2d33f-1] regionserver.HStore(310): Store=a2b878baabc37b3076d648a696d2d33f/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:27,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-12 15:00:27,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-12 15:00:27,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:27,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 469de65a1b57113b3e7565d5f5bf8201; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69823685, jitterRate=0.04045398533344269}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:27,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:27,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 469de65a1b57113b3e7565d5f5bf8201: Running coprocessor pre-open hook at 1676214027907Writing region info on filesystem at 1676214027907Initializing all the Stores at 1676214027908 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214027908Cleaning up temporary data from old regions at 1676214027934 (+26 ms)Cleaning up detritus from prior splits at 1676214027934Running coprocessor post-open hooks at 1676214027938 (+4 ms)Region opened successfully at 1676214027938 2023-02-12 15:00:27,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e31ecbd7855a2e0bca06cbe75ecfc8fe; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65173611, jitterRate=-0.02883751690387726}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:27,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Running coprocessor pre-open hook at 1676214027906Writing region info on filesystem at 1676214027906Initializing all the Stores at 1676214027907 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214027907Cleaning up temporary data from old regions at 1676214027934 (+27 ms)Cleaning up detritus from prior splits at 1676214027934Running coprocessor post-open hooks at 1676214027938 (+4 ms)Region opened successfully at 1676214027938 2023-02-12 15:00:27,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe., pid=184, masterSystemTime=1676214027872 2023-02-12 15:00:27,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201., pid=186, masterSystemTime=1676214027873 2023-02-12 15:00:27,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:27,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:27,941 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:27,941 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027941"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214027941"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214027941"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214027941"}]},"ts":"1676214027941"} 2023-02-12 15:00:27,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:27,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:27,942 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:27,942 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214027942"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214027942"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214027942"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214027942"}]},"ts":"1676214027942"} 2023-02-12 15:00:27,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:27,945 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=177 2023-02-12 15:00:27,945 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=177, state=SUCCESS; OpenRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377 in 220 msec 2023-02-12 15:00:27,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-12 15:00:27,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened a2b878baabc37b3076d648a696d2d33f; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58973596, jitterRate=-0.12122493982315063}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:27,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for a2b878baabc37b3076d648a696d2d33f: Running coprocessor pre-open hook at 1676214027907Writing region info on filesystem at 1676214027907Initializing all the Stores at 1676214027908 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214027908Cleaning up temporary data from old regions at 1676214027938 (+30 ms)Cleaning up detritus from prior splits at 1676214027940 (+2 ms)Running coprocessor post-open hooks at 1676214027947 (+7 ms)Region opened successfully at 1676214027947 2023-02-12 15:00:27,947 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=176 2023-02-12 15:00:27,947 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=176, state=SUCCESS; OpenRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,38453,1676214004461 in 220 msec 2023-02-12 15:00:27,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f., pid=187, masterSystemTime=1676214027874 2023-02-12 15:00:27,948 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, ASSIGN in 385 msec 2023-02-12 15:00:27,949 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, ASSIGN in 387 msec 2023-02-12 15:00:27,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:27,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:27,950 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:27,950 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214027949"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214027949"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214027949"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214027949"}]},"ts":"1676214027949"} 2023-02-12 15:00:27,953 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=180 2023-02-12 15:00:27,953 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=180, state=SUCCESS; OpenRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,33845,1676214004423 in 226 msec 2023-02-12 15:00:27,954 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=180, resume processing ppid=175 2023-02-12 15:00:27,954 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, ASSIGN in 393 msec 2023-02-12 15:00:27,955 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214027955"}]},"ts":"1676214027955"} 2023-02-12 15:00:27,956 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=ENABLED in hbase:meta 2023-02-12 15:00:27,961 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676214007285 snapshot=name: "snaptb1-1676214007285" table: "testRestoreSnapshot-1676214007285" creation_time: 1676214016742 type: DISABLED version: 2 ttl: 0 ) in 522 msec 2023-02-12 15:00:28,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-12 15:00:28,197 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676214007285, procId: 175 completed 2023-02-12 15:00:28,198 DEBUG [Listener at localhost/38323] zookeeper.ReadOnlyZKClient(139): Connect 0x4ff51970 to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:28,206 DEBUG [Listener at localhost/38323] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c9ce563, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:28,208 DEBUG [hconnection-0x69705b98-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:28,209 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53056, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:28,212 DEBUG [hconnection-0x69705b98-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:28,213 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37158, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:28,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,418 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 15:00:28,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,632 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 15:00:28,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,807 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 15:00:28,810 DEBUG [hconnection-0x69705b98-shared-pool-3] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:28,811 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53480, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:28,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:28,995 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 15:00:29,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,205 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 15:00:29,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,443 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1550b85e to 127.0.0.1:61893 2023-02-12 15:00:29,443 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:29,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:29,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:30,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:31,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:31,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:31,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:31,014 INFO [Listener at localhost/38323] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676214007285 2023-02-12 15:00:31,014 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676214007285 2023-02-12 15:00:31,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:31,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-12 15:00:31,018 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214031017"}]},"ts":"1676214031017"} 2023-02-12 15:00:31,019 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLING in hbase:meta 2023-02-12 15:00:31,022 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676214007285 to state=DISABLING 2023-02-12 15:00:31,022 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN}] 2023-02-12 15:00:31,024 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN 2023-02-12 15:00:31,024 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN 2023-02-12 15:00:31,024 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN 2023-02-12 15:00:31,024 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN 2023-02-12 15:00:31,024 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN 2023-02-12 15:00:31,025 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:31,025 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:31,025 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214031025"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214031025"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214031025"}]},"ts":"1676214031025"} 2023-02-12 15:00:31,025 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:31,025 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:31,025 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:31,025 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214031025"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214031025"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214031025"}]},"ts":"1676214031025"} 2023-02-12 15:00:31,025 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214031025"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214031025"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214031025"}]},"ts":"1676214031025"} 2023-02-12 15:00:31,025 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214031025"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214031025"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214031025"}]},"ts":"1676214031025"} 2023-02-12 15:00:31,025 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214031025"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214031025"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214031025"}]},"ts":"1676214031025"} 2023-02-12 15:00:31,026 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=191, state=RUNNABLE; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:31,027 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=194, state=RUNNABLE; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:31,028 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=190, state=RUNNABLE; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:31,031 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=192, state=RUNNABLE; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:31,031 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=193, state=RUNNABLE; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:31,032 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN 2023-02-12 15:00:31,032 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:31,032 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214031032"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214031032"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214031032"}]},"ts":"1676214031032"} 2023-02-12 15:00:31,033 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=189, state=RUNNABLE; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:31,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:31,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:31,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e116768b6cb1e8fb57dfa7c68914d39c, disabling compactions & flushes 2023-02-12 15:00:31,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 469de65a1b57113b3e7565d5f5bf8201, disabling compactions & flushes 2023-02-12 15:00:31,180 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:31,180 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:31,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:31,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:31,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. after waiting 0 ms 2023-02-12 15:00:31,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:31,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. after waiting 0 ms 2023-02-12 15:00:31,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:31,185 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:31,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing a2b878baabc37b3076d648a696d2d33f, disabling compactions & flushes 2023-02-12 15:00:31,186 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:31,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:31,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. after waiting 0 ms 2023-02-12 15:00:31,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:31,191 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-12 15:00:31,191 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-12 15:00:31,191 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-12 15:00:31,192 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201. 2023-02-12 15:00:31,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 469de65a1b57113b3e7565d5f5bf8201: Waiting for close lock at 1676214031180Running coprocessor pre-close hooks at 1676214031180Disabling compacts and flushes for region at 1676214031180Disabling writes for close at 1676214031181 (+1 ms)Writing region close event to WAL at 1676214031185 (+4 ms)Running coprocessor post-close hooks at 1676214031191 (+6 ms)Closed at 1676214031191 2023-02-12 15:00:31,192 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c. 2023-02-12 15:00:31,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e116768b6cb1e8fb57dfa7c68914d39c: Waiting for close lock at 1676214031179Running coprocessor pre-close hooks at 1676214031179Disabling compacts and flushes for region at 1676214031179Disabling writes for close at 1676214031181 (+2 ms)Writing region close event to WAL at 1676214031185 (+4 ms)Running coprocessor post-close hooks at 1676214031192 (+7 ms)Closed at 1676214031192 2023-02-12 15:00:31,192 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f. 2023-02-12 15:00:31,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for a2b878baabc37b3076d648a696d2d33f: Waiting for close lock at 1676214031185Running coprocessor pre-close hooks at 1676214031185Disabling compacts and flushes for region at 1676214031185Disabling writes for close at 1676214031186 (+1 ms)Writing region close event to WAL at 1676214031189 (+3 ms)Running coprocessor post-close hooks at 1676214031192 (+3 ms)Closed at 1676214031192 2023-02-12 15:00:31,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:31,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:31,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c70c08e0263941ae1bfbb2d57f1c3052, disabling compactions & flushes 2023-02-12 15:00:31,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:31,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:31,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. after waiting 0 ms 2023-02-12 15:00:31,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:31,194 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=469de65a1b57113b3e7565d5f5bf8201, regionState=CLOSED 2023-02-12 15:00:31,194 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214031194"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214031194"}]},"ts":"1676214031194"} 2023-02-12 15:00:31,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:31,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:31,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e31ecbd7855a2e0bca06cbe75ecfc8fe, disabling compactions & flushes 2023-02-12 15:00:31,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:31,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:31,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. after waiting 0 ms 2023-02-12 15:00:31,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:31,201 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=e116768b6cb1e8fb57dfa7c68914d39c, regionState=CLOSED 2023-02-12 15:00:31,201 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676214031201"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214031201"}]},"ts":"1676214031201"} 2023-02-12 15:00:31,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:31,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:31,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 87a15dc17c09c51160706b761e764ba2, disabling compactions & flushes 2023-02-12 15:00:31,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:31,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:31,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. after waiting 0 ms 2023-02-12 15:00:31,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:31,202 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=a2b878baabc37b3076d648a696d2d33f, regionState=CLOSED 2023-02-12 15:00:31,202 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214031202"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214031202"}]},"ts":"1676214031202"} 2023-02-12 15:00:31,205 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-12 15:00:31,206 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=189 2023-02-12 15:00:31,206 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-12 15:00:31,206 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=189, state=SUCCESS; CloseRegionProcedure 469de65a1b57113b3e7565d5f5bf8201, server=jenkins-hbase4.apache.org,38453,1676214004461 in 169 msec 2023-02-12 15:00:31,206 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052. 2023-02-12 15:00:31,206 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c70c08e0263941ae1bfbb2d57f1c3052: Waiting for close lock at 1676214031193Running coprocessor pre-close hooks at 1676214031193Disabling compacts and flushes for region at 1676214031193Disabling writes for close at 1676214031194 (+1 ms)Writing region close event to WAL at 1676214031201 (+7 ms)Running coprocessor post-close hooks at 1676214031206 (+5 ms)Closed at 1676214031206 2023-02-12 15:00:31,206 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe. 2023-02-12 15:00:31,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e31ecbd7855a2e0bca06cbe75ecfc8fe: Waiting for close lock at 1676214031200Running coprocessor pre-close hooks at 1676214031200Disabling compacts and flushes for region at 1676214031200Disabling writes for close at 1676214031201 (+1 ms)Writing region close event to WAL at 1676214031203 (+2 ms)Running coprocessor post-close hooks at 1676214031206 (+3 ms)Closed at 1676214031206 2023-02-12 15:00:31,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-12 15:00:31,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2. 2023-02-12 15:00:31,209 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 87a15dc17c09c51160706b761e764ba2: Waiting for close lock at 1676214031201Running coprocessor pre-close hooks at 1676214031201Disabling compacts and flushes for region at 1676214031201Disabling writes for close at 1676214031202 (+1 ms)Writing region close event to WAL at 1676214031205 (+3 ms)Running coprocessor post-close hooks at 1676214031208 (+3 ms)Closed at 1676214031209 (+1 ms) 2023-02-12 15:00:31,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:31,213 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=194 2023-02-12 15:00:31,213 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=193 2023-02-12 15:00:31,213 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=194, state=SUCCESS; CloseRegionProcedure e116768b6cb1e8fb57dfa7c68914d39c, server=jenkins-hbase4.apache.org,37113,1676214004377 in 176 msec 2023-02-12 15:00:31,213 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=193, state=SUCCESS; CloseRegionProcedure a2b878baabc37b3076d648a696d2d33f, server=jenkins-hbase4.apache.org,33845,1676214004423 in 173 msec 2023-02-12 15:00:31,213 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=469de65a1b57113b3e7565d5f5bf8201, UNASSIGN in 184 msec 2023-02-12 15:00:31,213 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=c70c08e0263941ae1bfbb2d57f1c3052, regionState=CLOSED 2023-02-12 15:00:31,213 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214031213"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214031213"}]},"ts":"1676214031213"} 2023-02-12 15:00:31,213 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:31,214 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=e31ecbd7855a2e0bca06cbe75ecfc8fe, regionState=CLOSED 2023-02-12 15:00:31,214 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214031214"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214031214"}]},"ts":"1676214031214"} 2023-02-12 15:00:31,214 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=a2b878baabc37b3076d648a696d2d33f, UNASSIGN in 191 msec 2023-02-12 15:00:31,214 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:31,214 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e116768b6cb1e8fb57dfa7c68914d39c, UNASSIGN in 191 msec 2023-02-12 15:00:31,215 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=87a15dc17c09c51160706b761e764ba2, regionState=CLOSED 2023-02-12 15:00:31,215 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676214031215"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214031215"}]},"ts":"1676214031215"} 2023-02-12 15:00:31,216 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=191 2023-02-12 15:00:31,216 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=191, state=SUCCESS; CloseRegionProcedure c70c08e0263941ae1bfbb2d57f1c3052, server=jenkins-hbase4.apache.org,38453,1676214004461 in 188 msec 2023-02-12 15:00:31,217 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=190 2023-02-12 15:00:31,217 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=190, state=SUCCESS; CloseRegionProcedure e31ecbd7855a2e0bca06cbe75ecfc8fe, server=jenkins-hbase4.apache.org,37113,1676214004377 in 187 msec 2023-02-12 15:00:31,217 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=c70c08e0263941ae1bfbb2d57f1c3052, UNASSIGN in 194 msec 2023-02-12 15:00:31,217 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=192 2023-02-12 15:00:31,217 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=192, state=SUCCESS; CloseRegionProcedure 87a15dc17c09c51160706b761e764ba2, server=jenkins-hbase4.apache.org,33845,1676214004423 in 188 msec 2023-02-12 15:00:31,218 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=e31ecbd7855a2e0bca06cbe75ecfc8fe, UNASSIGN in 195 msec 2023-02-12 15:00:31,219 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=192, resume processing ppid=188 2023-02-12 15:00:31,219 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676214007285, region=87a15dc17c09c51160706b761e764ba2, UNASSIGN in 195 msec 2023-02-12 15:00:31,219 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214031219"}]},"ts":"1676214031219"} 2023-02-12 15:00:31,220 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676214007285, state=DISABLED in hbase:meta 2023-02-12 15:00:31,222 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676214007285 to state=DISABLED 2023-02-12 15:00:31,223 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676214007285 in 208 msec 2023-02-12 15:00:31,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-12 15:00:31,269 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 188 completed 2023-02-12 15:00:31,270 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676214007285 2023-02-12 15:00:31,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:31,272 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:31,272 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:31,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-12 15:00:31,279 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:31,279 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:31,279 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:31,279 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:31,279 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:31,279 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:31,280 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits] 2023-02-12 15:00:31,280 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits] 2023-02-12 15:00:31,281 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits] 2023-02-12 15:00:31,281 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits] 2023-02-12 15:00:31,281 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits] 2023-02-12 15:00:31,281 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits] 2023-02-12 15:00:31,284 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-984b06764f54429d98a03a4b3659e10a already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-984b06764f54429d98a03a4b3659e10a. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027172 currentMtime: 1676214027543 2023-02-12 15:00:31,285 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-49b9c7e6778343f982623f169324b842 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-49b9c7e6778343f982623f169324b842. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027172 currentMtime: 1676214027536 2023-02-12 15:00:31,286 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-984b06764f54429d98a03a4b3659e10a to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-984b06764f54429d98a03a4b3659e10a.1676214031280. 2023-02-12 15:00:31,286 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-49b9c7e6778343f982623f169324b842 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-49b9c7e6778343f982623f169324b842.1676214031281. 2023-02-12 15:00:31,287 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-3a1b76d129894ec2adcd309280852eb4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-3a1b76d129894ec2adcd309280852eb4. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027172 currentMtime: 1676214027525 2023-02-12 15:00:31,287 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-4a211d423a9d40e08d63ef96097f6015 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-4a211d423a9d40e08d63ef96097f6015. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027172 currentMtime: 1676214027532 2023-02-12 15:00:31,287 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-984b06764f54429d98a03a4b3659e10a to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-984b06764f54429d98a03a4b3659e10a 2023-02-12 15:00:31,288 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-3a1b76d129894ec2adcd309280852eb4 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-3a1b76d129894ec2adcd309280852eb4.1676214031280. 2023-02-12 15:00:31,288 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-4a211d423a9d40e08d63ef96097f6015 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-4a211d423a9d40e08d63ef96097f6015.1676214031281. 2023-02-12 15:00:31,288 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-49b9c7e6778343f982623f169324b842 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-49b9c7e6778343f982623f169324b842 2023-02-12 15:00:31,288 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-393173ee6d644dfbb763f3ba123cdb44 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-393173ee6d644dfbb763f3ba123cdb44. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027172 currentMtime: 1676214027541 2023-02-12 15:00:31,289 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-9837fd0f2c07404bbfa672c073f42881 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-9837fd0f2c07404bbfa672c073f42881. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027172 currentMtime: 1676214027534 2023-02-12 15:00:31,289 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-3a1b76d129894ec2adcd309280852eb4 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-3a1b76d129894ec2adcd309280852eb4 2023-02-12 15:00:31,290 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-393173ee6d644dfbb763f3ba123cdb44 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-393173ee6d644dfbb763f3ba123cdb44.1676214031281. 2023-02-12 15:00:31,290 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-4a211d423a9d40e08d63ef96097f6015 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-4a211d423a9d40e08d63ef96097f6015 2023-02-12 15:00:31,290 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-9837fd0f2c07404bbfa672c073f42881 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-9837fd0f2c07404bbfa672c073f42881.1676214031281. 2023-02-12 15:00:31,291 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-e0edae35a0ff4d6db2a54417a26dc5a6 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-e0edae35a0ff4d6db2a54417a26dc5a6. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027175 currentMtime: 1676214027545 2023-02-12 15:00:31,291 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-e0edae35a0ff4d6db2a54417a26dc5a6 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-e0edae35a0ff4d6db2a54417a26dc5a6.1676214031280. 2023-02-12 15:00:31,292 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-b00b668a6b014eb4a4e4c4bb861069f4 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-b00b668a6b014eb4a4e4c4bb861069f4. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027174 currentMtime: 1676214027540 2023-02-12 15:00:31,292 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-393173ee6d644dfbb763f3ba123cdb44 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-393173ee6d644dfbb763f3ba123cdb44 2023-02-12 15:00:31,292 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-9837fd0f2c07404bbfa672c073f42881 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-9837fd0f2c07404bbfa672c073f42881 2023-02-12 15:00:31,292 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-a67854a9941340869b744861955b1505 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-a67854a9941340869b744861955b1505. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027174 currentMtime: 1676214027536 2023-02-12 15:00:31,292 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-b00b668a6b014eb4a4e4c4bb861069f4 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-b00b668a6b014eb4a4e4c4bb861069f4.1676214031281. 2023-02-12 15:00:31,293 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-498d9c0eb3304fc7aedc40e7676473e2 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-498d9c0eb3304fc7aedc40e7676473e2. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027175 currentMtime: 1676214027540 2023-02-12 15:00:31,293 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-a67854a9941340869b744861955b1505 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-a67854a9941340869b744861955b1505.1676214031281. 2023-02-12 15:00:31,293 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-e0edae35a0ff4d6db2a54417a26dc5a6 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/cf/testRestoreSnapshot-1676214007285=469de65a1b57113b3e7565d5f5bf8201-e0edae35a0ff4d6db2a54417a26dc5a6 2023-02-12 15:00:31,293 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-498d9c0eb3304fc7aedc40e7676473e2 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-498d9c0eb3304fc7aedc40e7676473e2.1676214031280. 2023-02-12 15:00:31,294 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-b00b668a6b014eb4a4e4c4bb861069f4 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/cf/testRestoreSnapshot-1676214007285=e116768b6cb1e8fb57dfa7c68914d39c-b00b668a6b014eb4a4e4c4bb861069f4 2023-02-12 15:00:31,295 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-a67854a9941340869b744861955b1505 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/cf/testRestoreSnapshot-1676214007285=e31ecbd7855a2e0bca06cbe75ecfc8fe-a67854a9941340869b744861955b1505 2023-02-12 15:00:31,295 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-f5c092a8a2e84ffe91bb687f3ea1c42d already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-f5c092a8a2e84ffe91bb687f3ea1c42d. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027174 currentMtime: 1676214027541 2023-02-12 15:00:31,295 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-498d9c0eb3304fc7aedc40e7676473e2 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/cf/testRestoreSnapshot-1676214007285=c70c08e0263941ae1bfbb2d57f1c3052-498d9c0eb3304fc7aedc40e7676473e2 2023-02-12 15:00:31,296 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-5718cdbb97f24b3081185dfde2fb1e55 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-5718cdbb97f24b3081185dfde2fb1e55. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027174 currentMtime: 1676214027544 2023-02-12 15:00:31,296 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-f5c092a8a2e84ffe91bb687f3ea1c42d to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-f5c092a8a2e84ffe91bb687f3ea1c42d.1676214031281. 2023-02-12 15:00:31,297 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-5718cdbb97f24b3081185dfde2fb1e55 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-5718cdbb97f24b3081185dfde2fb1e55.1676214031281. 2023-02-12 15:00:31,298 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-f5c092a8a2e84ffe91bb687f3ea1c42d to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/cf/testRestoreSnapshot-1676214007285=a2b878baabc37b3076d648a696d2d33f-f5c092a8a2e84ffe91bb687f3ea1c42d 2023-02-12 15:00:31,298 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits/18.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c/recovered.edits/18.seqid 2023-02-12 15:00:31,299 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-5718cdbb97f24b3081185dfde2fb1e55 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/cf/testRestoreSnapshot-1676214007285=87a15dc17c09c51160706b761e764ba2-5718cdbb97f24b3081185dfde2fb1e55 2023-02-12 15:00:31,299 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits/18.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201/recovered.edits/18.seqid 2023-02-12 15:00:31,299 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e116768b6cb1e8fb57dfa7c68914d39c 2023-02-12 15:00:31,299 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/469de65a1b57113b3e7565d5f5bf8201 2023-02-12 15:00:31,300 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits/18.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052/recovered.edits/18.seqid 2023-02-12 15:00:31,301 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/c70c08e0263941ae1bfbb2d57f1c3052 2023-02-12 15:00:31,301 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits/18.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe/recovered.edits/18.seqid 2023-02-12 15:00:31,302 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/e31ecbd7855a2e0bca06cbe75ecfc8fe 2023-02-12 15:00:31,303 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits/18.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f/recovered.edits/18.seqid 2023-02-12 15:00:31,303 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits/18.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2/recovered.edits/18.seqid 2023-02-12 15:00:31,303 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/a2b878baabc37b3076d648a696d2d33f 2023-02-12 15:00:31,303 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testRestoreSnapshot-1676214007285/87a15dc17c09c51160706b761e764ba2 2023-02-12 15:00:31,303 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676214007285 regions 2023-02-12 15:00:31,304 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2 2023-02-12 15:00:31,305 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf] 2023-02-12 15:00:31,308 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027184 currentMtime: 1676214027460 2023-02-12 15:00:31,309 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2.1676214031305. 2023-02-12 15:00:31,309 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c2023021279a7cb3e47534b189234a55689ed5ce2 2023-02-12 15:00:31,310 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027185 currentMtime: 1676214027462 2023-02-12 15:00:31,311 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95.1676214031305. 2023-02-12 15:00:31,312 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-a87ff679a2f3e71d9181a67b7542122c20230212cb0f881aca884dd8846d6d549857bd95 2023-02-12 15:00:31,313 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027186 currentMtime: 1676214027464 2023-02-12 15:00:31,314 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5.1676214031305. 2023-02-12 15:00:31,315 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302123f776bd3b5f94049833af169532b11c5 2023-02-12 15:00:31,316 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027187 currentMtime: 1676214027466 2023-02-12 15:00:31,317 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257.1676214031305. 2023-02-12 15:00:31,318 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c4ca4238a0b923820dcc509a6f75849b202302127be60bcbf240460f9449679cab442257 2023-02-12 15:00:31,319 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027188 currentMtime: 1676214027467 2023-02-12 15:00:31,319 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c.1676214031305. 2023-02-12 15:00:31,320 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302124ebddee338d744febafdb4ea4b06999c 2023-02-12 15:00:31,321 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027189 currentMtime: 1676214027469 2023-02-12 15:00:31,322 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898.1676214031305. 2023-02-12 15:00:31,322 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-c81e728d9d4c2f636f067f89cc14862c202302129a8aa1c39fa641c9842acd8ea8922898 2023-02-12 15:00:31,324 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027190 currentMtime: 1676214027471 2023-02-12 15:00:31,324 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99.1676214031305. 2023-02-12 15:00:31,325 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e202302126ac519f36ca9472c8c721e5e83391a99 2023-02-12 15:00:31,326 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027191 currentMtime: 1676214027472 2023-02-12 15:00:31,326 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b.1676214031305. 2023-02-12 15:00:31,327 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-d41d8cd98f00b204e9800998ecf8427e20230212e6324038930b4beaa09a290b8049587b 2023-02-12 15:00:31,328 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027192 currentMtime: 1676214027474 2023-02-12 15:00:31,328 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a.1676214031305. 2023-02-12 15:00:31,329 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212451b51fba3d0401cbcd1fcbd9b9dd60a 2023-02-12 15:00:31,330 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027193 currentMtime: 1676214027475 2023-02-12 15:00:31,330 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30.1676214031305. 2023-02-12 15:00:31,331 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-e4da3b7fbbce2345d7772b0674a318d520230212f311cb0d64294fbebeaf9e4bbeff7e30 2023-02-12 15:00:31,332 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027195 currentMtime: 1676214027477 2023-02-12 15:00:31,333 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766.1676214031305. 2023-02-12 15:00:31,333 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b189b53860c64b3eaf08d7f4f1f1c766 2023-02-12 15:00:31,334 ERROR [PEWorker-1] backup.HFileArchiver(523): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f. archiveLen: 0 currentLen: 0 archiveMtime: 1676214027196 currentMtime: 1676214027479 2023-02-12 15:00:31,335 INFO [PEWorker-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f.1676214031305. 2023-02-12 15:00:31,335 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2/cf/testRestoreSnapshot-1676214007285=e793c55fa93954c06e9823103f4574b2-eccbc87e4b5ce2fe28308fd9f2a7baf320230212b9826d7dcfdd40ec9a5bc26795b38d8f 2023-02-12 15:00:31,336 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testRestoreSnapshot-1676214007285/e793c55fa93954c06e9823103f4574b2 2023-02-12 15:00:31,338 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:31,341 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676214007285 from hbase:meta 2023-02-12 15:00:31,344 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676214007285' descriptor. 2023-02-12 15:00:31,345 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:31,345 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676214007285' from region states. 2023-02-12 15:00:31,345 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214031345"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:31,345 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214031345"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:31,345 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214031345"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:31,345 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214031345"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:31,345 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214031345"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:31,345 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214031345"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:31,347 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-12 15:00:31,347 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 469de65a1b57113b3e7565d5f5bf8201, NAME => 'testRestoreSnapshot-1676214007285,,1676214007309.469de65a1b57113b3e7565d5f5bf8201.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e31ecbd7855a2e0bca06cbe75ecfc8fe, NAME => 'testRestoreSnapshot-1676214007285,1,1676214007309.e31ecbd7855a2e0bca06cbe75ecfc8fe.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => c70c08e0263941ae1bfbb2d57f1c3052, NAME => 'testRestoreSnapshot-1676214007285,2,1676214007309.c70c08e0263941ae1bfbb2d57f1c3052.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 87a15dc17c09c51160706b761e764ba2, NAME => 'testRestoreSnapshot-1676214007285,3,1676214007309.87a15dc17c09c51160706b761e764ba2.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => a2b878baabc37b3076d648a696d2d33f, NAME => 'testRestoreSnapshot-1676214007285,4,1676214007309.a2b878baabc37b3076d648a696d2d33f.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => e116768b6cb1e8fb57dfa7c68914d39c, NAME => 'testRestoreSnapshot-1676214007285,5,1676214007309.e116768b6cb1e8fb57dfa7c68914d39c.', STARTKEY => '5', ENDKEY => ''}] 2023-02-12 15:00:31,347 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676214007285' as deleted. 2023-02-12 15:00:31,347 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676214007285","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676214031347"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:31,348 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676214007285 state from META 2023-02-12 15:00:31,351 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 2023-02-12 15:00:31,352 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676214007285 in 81 msec 2023-02-12 15:00:31,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-12 15:00:31,524 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676214007285, procId: 201 completed 2023-02-12 15:00:31,529 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676214007285" 2023-02-12 15:00:31,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676214007285 2023-02-12 15:00:31,532 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676214007285" 2023-02-12 15:00:31,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676214007285 2023-02-12 15:00:31,534 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb1-1676214007285" 2023-02-12 15:00:31,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676214007285 2023-02-12 15:00:31,562 INFO [Listener at localhost/38323] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=426 (was 361) Potentially hanging thread: hconnection-0x7b9b1635-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_336938306_15 at /127.0.0.1:34998 [Waiting for operation #479] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1328563048_15 at /127.0.0.1:42006 [Waiting for operation #513] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:61893@0x4ff51970-SendThread(127.0.0.1:61893) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1328563048_15 at /127.0.0.1:41996 [Waiting for operation #511] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x779b93d1-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-147340794_15 at /127.0.0.1:40680 [Waiting for operation #1322] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_336938306_15 at /127.0.0.1:41980 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x779b93d1-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x779b93d1-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:61893@0x4ff51970 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/475763653.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-147340794_15 at /127.0.0.1:58354 [Waiting for operation #535] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_336938306_15 at /127.0.0.1:41984 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1328563048_15 at /127.0.0.1:45644 [Waiting for operation #509] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_823717739_15 at /127.0.0.1:43962 [Waiting for operation #272] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-147340794_15 at /127.0.0.1:58928 [Waiting for operation #509] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1328563048_15 at /127.0.0.1:45658 [Waiting for operation #479] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1328563048_15 at /127.0.0.1:58062 [Waiting for operation #562] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-metaLookup-shared--pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:61893@0x4ff51970-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_336938306_15 at /127.0.0.1:45642 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x69705b98-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_823717739_15 at /127.0.0.1:58930 [Waiting for operation #517] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1328563048_15 at /127.0.0.1:43954 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_823717739_15 at /127.0.0.1:59036 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x779b93d1-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=826 (was 695) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=186 (was 211), ProcessCount=168 (was 171), AvailableMemoryMB=3593 (was 3907) 2023-02-12 15:00:31,581 INFO [Listener at localhost/38323] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=426, OpenFileDescriptor=826, MaxFileDescriptor=60000, SystemLoadAverage=186, ProcessCount=168, AvailableMemoryMB=3593 2023-02-12 15:00:31,584 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testCorruptedSnapshot-1676214031582', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-12 15:00:31,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676214031582 2023-02-12 15:00:31,586 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676214031582 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-12 15:00:31,586 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676214031582" procId is: 202 2023-02-12 15:00:31,587 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676214031582 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-12 15:00:31,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-12 15:00:31,590 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:31,590 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:31,590 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:31,590 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:31,590 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:31,590 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:31,591 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7 empty. 2023-02-12 15:00:31,591 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3 empty. 2023-02-12 15:00:31,591 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d empty. 2023-02-12 15:00:31,591 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c empty. 2023-02-12 15:00:31,591 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150 empty. 2023-02-12 15:00:31,592 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:31,592 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:31,592 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471 empty. 2023-02-12 15:00:31,592 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:31,592 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:31,592 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:31,592 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:31,592 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676214031582 regions 2023-02-12 15:00:31,604 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/.tabledesc/.tableinfo.0000000001 2023-02-12 15:00:31,606 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(7671): creating {ENCODED => 629e2ec9e8223cc0fb7f0875b15cef8c, NAME => 'testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676214031582', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:31,606 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(7671): creating {ENCODED => e5e89645c25979bd82f9e1e324b861b7, NAME => 'testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676214031582', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:31,606 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(7671): creating {ENCODED => 1886cfbba21a42a84ae46d52d88f5150, NAME => 'testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676214031582', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:31,624 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:31,624 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:31,624 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1603): Closing e5e89645c25979bd82f9e1e324b861b7, disabling compactions & flushes 2023-02-12 15:00:31,624 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1603): Closing 629e2ec9e8223cc0fb7f0875b15cef8c, disabling compactions & flushes 2023-02-12 15:00:31,624 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:31,624 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:31,624 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:31,624 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:31,624 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. after waiting 0 ms 2023-02-12 15:00:31,624 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. after waiting 0 ms 2023-02-12 15:00:31,624 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:31,624 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:31,625 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:31,625 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:31,625 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1557): Region close journal for e5e89645c25979bd82f9e1e324b861b7: Waiting for close lock at 1676214031624Disabling compacts and flushes for region at 1676214031624Disabling writes for close at 1676214031624Writing region close event to WAL at 1676214031625 (+1 ms)Closed at 1676214031625 2023-02-12 15:00:31,625 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1557): Region close journal for 629e2ec9e8223cc0fb7f0875b15cef8c: Waiting for close lock at 1676214031624Disabling compacts and flushes for region at 1676214031624Disabling writes for close at 1676214031624Writing region close event to WAL at 1676214031625 (+1 ms)Closed at 1676214031625 2023-02-12 15:00:31,625 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(7671): creating {ENCODED => c54693fd62fafde40475f3c394bfbc2d, NAME => 'testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676214031582', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:31,625 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(7671): creating {ENCODED => 7736b0c91a68e683d42e81f85bf8f471, NAME => 'testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676214031582', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:31,661 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:31,661 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1603): Closing 1886cfbba21a42a84ae46d52d88f5150, disabling compactions & flushes 2023-02-12 15:00:31,661 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:31,661 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:31,661 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. after waiting 0 ms 2023-02-12 15:00:31,661 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:31,662 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:31,662 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1557): Region close journal for 1886cfbba21a42a84ae46d52d88f5150: Waiting for close lock at 1676214031661Disabling compacts and flushes for region at 1676214031661Disabling writes for close at 1676214031661Writing region close event to WAL at 1676214031661Closed at 1676214031662 (+1 ms) 2023-02-12 15:00:31,662 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(7671): creating {ENCODED => 6caf9978ca0972f75c0854f1c9fb76e3, NAME => 'testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676214031582', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp 2023-02-12 15:00:31,664 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:31,664 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1603): Closing 7736b0c91a68e683d42e81f85bf8f471, disabling compactions & flushes 2023-02-12 15:00:31,664 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:31,664 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:31,664 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. after waiting 0 ms 2023-02-12 15:00:31,664 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:31,664 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:31,664 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-2] regionserver.HRegion(1557): Region close journal for 7736b0c91a68e683d42e81f85bf8f471: Waiting for close lock at 1676214031664Disabling compacts and flushes for region at 1676214031664Disabling writes for close at 1676214031664Writing region close event to WAL at 1676214031664Closed at 1676214031664 2023-02-12 15:00:31,668 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:31,668 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1603): Closing c54693fd62fafde40475f3c394bfbc2d, disabling compactions & flushes 2023-02-12 15:00:31,668 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:31,668 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:31,668 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. after waiting 0 ms 2023-02-12 15:00:31,668 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:31,668 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:31,668 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-0] regionserver.HRegion(1557): Region close journal for c54693fd62fafde40475f3c394bfbc2d: Waiting for close lock at 1676214031668Disabling compacts and flushes for region at 1676214031668Disabling writes for close at 1676214031668Writing region close event to WAL at 1676214031668Closed at 1676214031668 2023-02-12 15:00:31,672 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:31,672 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1603): Closing 6caf9978ca0972f75c0854f1c9fb76e3, disabling compactions & flushes 2023-02-12 15:00:31,672 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:31,672 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:31,672 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. after waiting 0 ms 2023-02-12 15:00:31,672 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:31,672 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:31,672 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676214031582-pool-1] regionserver.HRegion(1557): Region close journal for 6caf9978ca0972f75c0854f1c9fb76e3: Waiting for close lock at 1676214031672Disabling compacts and flushes for region at 1676214031672Disabling writes for close at 1676214031672Writing region close event to WAL at 1676214031672Closed at 1676214031672 2023-02-12 15:00:31,674 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676214031582 execute state=CREATE_TABLE_ADD_TO_META 2023-02-12 15:00:31,675 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214031675"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214031675"}]},"ts":"1676214031675"} 2023-02-12 15:00:31,675 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214031675"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214031675"}]},"ts":"1676214031675"} 2023-02-12 15:00:31,675 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214031675"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214031675"}]},"ts":"1676214031675"} 2023-02-12 15:00:31,675 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214031675"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214031675"}]},"ts":"1676214031675"} 2023-02-12 15:00:31,675 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214031675"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214031675"}]},"ts":"1676214031675"} 2023-02-12 15:00:31,675 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214031675"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214031675"}]},"ts":"1676214031675"} 2023-02-12 15:00:31,680 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-12 15:00:31,680 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676214031582 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-12 15:00:31,681 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214031681"}]},"ts":"1676214031681"} 2023-02-12 15:00:31,682 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676214031582, state=ENABLING in hbase:meta 2023-02-12 15:00:31,686 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-12 15:00:31,688 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-12 15:00:31,688 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-12 15:00:31,688 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-12 15:00:31,688 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-12 15:00:31,689 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, ASSIGN}] 2023-02-12 15:00:31,691 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, ASSIGN 2023-02-12 15:00:31,691 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, ASSIGN 2023-02-12 15:00:31,691 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, ASSIGN 2023-02-12 15:00:31,691 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, ASSIGN 2023-02-12 15:00:31,692 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=false 2023-02-12 15:00:31,692 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=false 2023-02-12 15:00:31,692 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=false 2023-02-12 15:00:31,692 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=false 2023-02-12 15:00:31,692 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, ASSIGN 2023-02-12 15:00:31,693 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, ASSIGN 2023-02-12 15:00:31,693 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=false 2023-02-12 15:00:31,693 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=false 2023-02-12 15:00:31,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-12 15:00:31,842 INFO [jenkins-hbase4:45369] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 15:00:31,845 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=6caf9978ca0972f75c0854f1c9fb76e3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:31,845 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=c54693fd62fafde40475f3c394bfbc2d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:31,845 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214031845"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214031845"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214031845"}]},"ts":"1676214031845"} 2023-02-12 15:00:31,845 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=629e2ec9e8223cc0fb7f0875b15cef8c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:31,845 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=7736b0c91a68e683d42e81f85bf8f471, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:31,845 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214031845"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214031845"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214031845"}]},"ts":"1676214031845"} 2023-02-12 15:00:31,845 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=1886cfbba21a42a84ae46d52d88f5150, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:31,845 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214031845"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214031845"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214031845"}]},"ts":"1676214031845"} 2023-02-12 15:00:31,846 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214031845"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214031845"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214031845"}]},"ts":"1676214031845"} 2023-02-12 15:00:31,845 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214031845"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214031845"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214031845"}]},"ts":"1676214031845"} 2023-02-12 15:00:31,847 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=208, state=RUNNABLE; OpenRegionProcedure 6caf9978ca0972f75c0854f1c9fb76e3, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:31,847 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=205, state=RUNNABLE; OpenRegionProcedure 629e2ec9e8223cc0fb7f0875b15cef8c, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:31,848 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=207, state=RUNNABLE; OpenRegionProcedure 7736b0c91a68e683d42e81f85bf8f471, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:31,848 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=204, state=RUNNABLE; OpenRegionProcedure 1886cfbba21a42a84ae46d52d88f5150, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:31,849 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=e5e89645c25979bd82f9e1e324b861b7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:31,849 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=206, state=RUNNABLE; OpenRegionProcedure c54693fd62fafde40475f3c394bfbc2d, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:31,849 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214031849"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214031849"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214031849"}]},"ts":"1676214031849"} 2023-02-12 15:00:31,850 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=203, state=RUNNABLE; OpenRegionProcedure e5e89645c25979bd82f9e1e324b861b7, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:32,001 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:32,001 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:32,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c54693fd62fafde40475f3c394bfbc2d, NAME => 'testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 15:00:32,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6caf9978ca0972f75c0854f1c9fb76e3, NAME => 'testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 15:00:32,002 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:32,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e5e89645c25979bd82f9e1e324b861b7, NAME => 'testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 15:00:32,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:32,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:32,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:32,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:32,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:32,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:32,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:32,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:32,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:32,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:32,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:32,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:32,003 INFO [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:32,003 INFO [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:32,003 INFO [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:32,004 DEBUG [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf 2023-02-12 15:00:32,004 DEBUG [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf 2023-02-12 15:00:32,004 DEBUG [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf 2023-02-12 15:00:32,005 DEBUG [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf 2023-02-12 15:00:32,005 DEBUG [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf 2023-02-12 15:00:32,005 DEBUG [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf 2023-02-12 15:00:32,005 INFO [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c54693fd62fafde40475f3c394bfbc2d columnFamilyName cf 2023-02-12 15:00:32,005 INFO [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6caf9978ca0972f75c0854f1c9fb76e3 columnFamilyName cf 2023-02-12 15:00:32,005 INFO [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e5e89645c25979bd82f9e1e324b861b7 columnFamilyName cf 2023-02-12 15:00:32,006 INFO [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] regionserver.HStore(310): Store=6caf9978ca0972f75c0854f1c9fb76e3/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:32,006 INFO [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] regionserver.HStore(310): Store=c54693fd62fafde40475f3c394bfbc2d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:32,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:32,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:32,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:32,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:32,008 INFO [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] regionserver.HStore(310): Store=e5e89645c25979bd82f9e1e324b861b7/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:32,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:32,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:32,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:32,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:32,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:32,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:32,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:32,012 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c54693fd62fafde40475f3c394bfbc2d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62403551, jitterRate=-0.0701146274805069}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:32,012 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 6caf9978ca0972f75c0854f1c9fb76e3; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71878705, jitterRate=0.0710761696100235}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:32,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c54693fd62fafde40475f3c394bfbc2d: Running coprocessor pre-open hook at 1676214032002Writing region info on filesystem at 1676214032002Initializing all the Stores at 1676214032003 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214032003Cleaning up temporary data from old regions at 1676214032008 (+5 ms)Cleaning up detritus from prior splits at 1676214032008Running coprocessor post-open hooks at 1676214032012 (+4 ms)Region opened successfully at 1676214032012 2023-02-12 15:00:32,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 6caf9978ca0972f75c0854f1c9fb76e3: Running coprocessor pre-open hook at 1676214032002Writing region info on filesystem at 1676214032002Initializing all the Stores at 1676214032003 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214032003Cleaning up temporary data from old regions at 1676214032008 (+5 ms)Cleaning up detritus from prior splits at 1676214032008Running coprocessor post-open hooks at 1676214032012 (+4 ms)Region opened successfully at 1676214032012 2023-02-12 15:00:32,013 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3., pid=209, masterSystemTime=1676214031998 2023-02-12 15:00:32,013 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d., pid=213, masterSystemTime=1676214031998 2023-02-12 15:00:32,014 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:32,014 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e5e89645c25979bd82f9e1e324b861b7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60995699, jitterRate=-0.09109325706958771}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:32,014 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e5e89645c25979bd82f9e1e324b861b7: Running coprocessor pre-open hook at 1676214032002Writing region info on filesystem at 1676214032002Initializing all the Stores at 1676214032003 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214032003Cleaning up temporary data from old regions at 1676214032010 (+7 ms)Cleaning up detritus from prior splits at 1676214032010Running coprocessor post-open hooks at 1676214032014 (+4 ms)Region opened successfully at 1676214032014 2023-02-12 15:00:32,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:32,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:32,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:32,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7., pid=214, masterSystemTime=1676214031999 2023-02-12 15:00:32,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1886cfbba21a42a84ae46d52d88f5150, NAME => 'testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 15:00:32,015 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=6caf9978ca0972f75c0854f1c9fb76e3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:32,015 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214032015"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214032015"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214032015"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214032015"}]},"ts":"1676214032015"} 2023-02-12 15:00:32,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:32,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:32,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:32,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:32,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:32,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:32,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:32,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 629e2ec9e8223cc0fb7f0875b15cef8c, NAME => 'testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 15:00:32,016 INFO [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:32,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:32,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:32,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:32,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:32,017 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=c54693fd62fafde40475f3c394bfbc2d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:32,017 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214032017"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214032017"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214032017"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214032017"}]},"ts":"1676214032017"} 2023-02-12 15:00:32,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:32,018 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:32,018 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:32,018 DEBUG [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf 2023-02-12 15:00:32,018 INFO [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:32,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7736b0c91a68e683d42e81f85bf8f471, NAME => 'testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 15:00:32,018 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=e5e89645c25979bd82f9e1e324b861b7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:32,018 DEBUG [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf 2023-02-12 15:00:32,018 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214032018"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214032018"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214032018"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214032018"}]},"ts":"1676214032018"} 2023-02-12 15:00:32,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:32,018 INFO [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1886cfbba21a42a84ae46d52d88f5150 columnFamilyName cf 2023-02-12 15:00:32,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:32,019 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:32,019 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:32,019 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=208 2023-02-12 15:00:32,019 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=208, state=SUCCESS; OpenRegionProcedure 6caf9978ca0972f75c0854f1c9fb76e3, server=jenkins-hbase4.apache.org,38453,1676214004461 in 170 msec 2023-02-12 15:00:32,019 DEBUG [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf 2023-02-12 15:00:32,019 INFO [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] regionserver.HStore(310): Store=1886cfbba21a42a84ae46d52d88f5150/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:32,019 DEBUG [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf 2023-02-12 15:00:32,020 INFO [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 629e2ec9e8223cc0fb7f0875b15cef8c columnFamilyName cf 2023-02-12 15:00:32,020 INFO [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:32,020 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:32,021 INFO [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] regionserver.HStore(310): Store=629e2ec9e8223cc0fb7f0875b15cef8c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:32,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:32,021 DEBUG [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf 2023-02-12 15:00:32,021 DEBUG [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf 2023-02-12 15:00:32,022 INFO [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7736b0c91a68e683d42e81f85bf8f471 columnFamilyName cf 2023-02-12 15:00:32,022 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, ASSIGN in 330 msec 2023-02-12 15:00:32,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:32,022 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:32,023 INFO [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] regionserver.HStore(310): Store=7736b0c91a68e683d42e81f85bf8f471/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:32,023 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=206 2023-02-12 15:00:32,023 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=206, state=SUCCESS; OpenRegionProcedure c54693fd62fafde40475f3c394bfbc2d, server=jenkins-hbase4.apache.org,37113,1676214004377 in 171 msec 2023-02-12 15:00:32,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:32,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:32,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:32,027 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=203 2023-02-12 15:00:32,027 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=203, state=SUCCESS; OpenRegionProcedure e5e89645c25979bd82f9e1e324b861b7, server=jenkins-hbase4.apache.org,33845,1676214004423 in 171 msec 2023-02-12 15:00:32,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:32,028 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, ASSIGN in 334 msec 2023-02-12 15:00:32,029 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, ASSIGN in 338 msec 2023-02-12 15:00:32,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:32,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:32,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 629e2ec9e8223cc0fb7f0875b15cef8c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68201274, jitterRate=0.016278177499771118}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:32,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1886cfbba21a42a84ae46d52d88f5150; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59435313, jitterRate=-0.11434482038021088}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:32,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 629e2ec9e8223cc0fb7f0875b15cef8c: Running coprocessor pre-open hook at 1676214032017Writing region info on filesystem at 1676214032017Initializing all the Stores at 1676214032017Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214032017Cleaning up temporary data from old regions at 1676214032023 (+6 ms)Cleaning up detritus from prior splits at 1676214032024 (+1 ms)Running coprocessor post-open hooks at 1676214032031 (+7 ms)Region opened successfully at 1676214032031 2023-02-12 15:00:32,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1886cfbba21a42a84ae46d52d88f5150: Running coprocessor pre-open hook at 1676214032015Writing region info on filesystem at 1676214032015Initializing all the Stores at 1676214032016 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214032016Cleaning up temporary data from old regions at 1676214032021 (+5 ms)Cleaning up detritus from prior splits at 1676214032022 (+1 ms)Running coprocessor post-open hooks at 1676214032031 (+9 ms)Region opened successfully at 1676214032031 2023-02-12 15:00:32,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:32,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-12 15:00:32,034 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7736b0c91a68e683d42e81f85bf8f471; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60738992, jitterRate=-0.09491848945617676}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:32,035 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7736b0c91a68e683d42e81f85bf8f471: Running coprocessor pre-open hook at 1676214032019Writing region info on filesystem at 1676214032019Initializing all the Stores at 1676214032019Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214032020 (+1 ms)Cleaning up temporary data from old regions at 1676214032027 (+7 ms)Cleaning up detritus from prior splits at 1676214032027Running coprocessor post-open hooks at 1676214032035 (+8 ms)Region opened successfully at 1676214032035 2023-02-12 15:00:32,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150., pid=212, masterSystemTime=1676214031998 2023-02-12 15:00:32,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c., pid=210, masterSystemTime=1676214031998 2023-02-12 15:00:32,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471., pid=211, masterSystemTime=1676214031999 2023-02-12 15:00:32,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:32,037 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:32,038 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=1886cfbba21a42a84ae46d52d88f5150, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:32,038 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214032038"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214032038"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214032038"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214032038"}]},"ts":"1676214032038"} 2023-02-12 15:00:32,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:32,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:32,038 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=7736b0c91a68e683d42e81f85bf8f471, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:32,038 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214032038"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214032038"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214032038"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214032038"}]},"ts":"1676214032038"} 2023-02-12 15:00:32,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:32,039 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:32,039 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=629e2ec9e8223cc0fb7f0875b15cef8c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:32,039 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214032039"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214032039"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214032039"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214032039"}]},"ts":"1676214032039"} 2023-02-12 15:00:32,041 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=204 2023-02-12 15:00:32,041 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=204, state=SUCCESS; OpenRegionProcedure 1886cfbba21a42a84ae46d52d88f5150, server=jenkins-hbase4.apache.org,38453,1676214004461 in 191 msec 2023-02-12 15:00:32,041 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=207 2023-02-12 15:00:32,041 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=207, state=SUCCESS; OpenRegionProcedure 7736b0c91a68e683d42e81f85bf8f471, server=jenkins-hbase4.apache.org,33845,1676214004423 in 192 msec 2023-02-12 15:00:32,043 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, ASSIGN in 352 msec 2023-02-12 15:00:32,043 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=205 2023-02-12 15:00:32,043 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=205, state=SUCCESS; OpenRegionProcedure 629e2ec9e8223cc0fb7f0875b15cef8c, server=jenkins-hbase4.apache.org,37113,1676214004377 in 194 msec 2023-02-12 15:00:32,043 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, ASSIGN in 352 msec 2023-02-12 15:00:32,044 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=205, resume processing ppid=202 2023-02-12 15:00:32,044 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, ASSIGN in 354 msec 2023-02-12 15:00:32,045 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676214031582 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-12 15:00:32,045 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214032045"}]},"ts":"1676214032045"} 2023-02-12 15:00:32,046 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676214031582, state=ENABLED in hbase:meta 2023-02-12 15:00:32,051 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676214031582 execute state=CREATE_TABLE_POST_OPERATION 2023-02-12 15:00:32,053 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676214031582 in 468 msec 2023-02-12 15:00:32,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-12 15:00:32,339 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676214031582, procId: 202 completed 2023-02-12 15:00:32,341 DEBUG [Listener at localhost/38323] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676214031582 2023-02-12 15:00:32,342 DEBUG [Listener at localhost/38323] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:32,344 INFO [Listener at localhost/38323] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-12 15:00:32,357 INFO [Listener at localhost/38323] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676214031582 2023-02-12 15:00:32,357 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676214031582 2023-02-12 15:00:32,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676214031582 2023-02-12 15:00:32,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-12 15:00:32,361 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214032361"}]},"ts":"1676214032361"} 2023-02-12 15:00:32,362 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676214031582, state=DISABLING in hbase:meta 2023-02-12 15:00:32,364 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676214031582 to state=DISABLING 2023-02-12 15:00:32,364 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, UNASSIGN}] 2023-02-12 15:00:32,366 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, UNASSIGN 2023-02-12 15:00:32,366 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, UNASSIGN 2023-02-12 15:00:32,366 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, UNASSIGN 2023-02-12 15:00:32,366 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, UNASSIGN 2023-02-12 15:00:32,366 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, UNASSIGN 2023-02-12 15:00:32,366 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=c54693fd62fafde40475f3c394bfbc2d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:32,367 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214032366"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214032366"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214032366"}]},"ts":"1676214032366"} 2023-02-12 15:00:32,367 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=e5e89645c25979bd82f9e1e324b861b7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:32,367 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=6caf9978ca0972f75c0854f1c9fb76e3, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:32,367 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214032367"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214032367"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214032367"}]},"ts":"1676214032367"} 2023-02-12 15:00:32,367 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=629e2ec9e8223cc0fb7f0875b15cef8c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:32,367 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=1886cfbba21a42a84ae46d52d88f5150, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:32,367 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214032367"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214032367"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214032367"}]},"ts":"1676214032367"} 2023-02-12 15:00:32,367 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214032367"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214032367"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214032367"}]},"ts":"1676214032367"} 2023-02-12 15:00:32,367 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214032367"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214032367"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214032367"}]},"ts":"1676214032367"} 2023-02-12 15:00:32,368 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=219, state=RUNNABLE; CloseRegionProcedure c54693fd62fafde40475f3c394bfbc2d, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:32,369 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=216, state=RUNNABLE; CloseRegionProcedure e5e89645c25979bd82f9e1e324b861b7, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:32,369 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=221, state=RUNNABLE; CloseRegionProcedure 6caf9978ca0972f75c0854f1c9fb76e3, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:32,371 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=217, state=RUNNABLE; CloseRegionProcedure 1886cfbba21a42a84ae46d52d88f5150, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:32,372 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, UNASSIGN 2023-02-12 15:00:32,372 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=218, state=RUNNABLE; CloseRegionProcedure 629e2ec9e8223cc0fb7f0875b15cef8c, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:32,373 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=7736b0c91a68e683d42e81f85bf8f471, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:32,373 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214032373"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214032373"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214032373"}]},"ts":"1676214032373"} 2023-02-12 15:00:32,374 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=220, state=RUNNABLE; CloseRegionProcedure 7736b0c91a68e683d42e81f85bf8f471, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:32,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:32,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c54693fd62fafde40475f3c394bfbc2d, disabling compactions & flushes 2023-02-12 15:00:32,521 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:32,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:32,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. after waiting 0 ms 2023-02-12 15:00:32,521 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:32,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:32,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:32,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1886cfbba21a42a84ae46d52d88f5150, disabling compactions & flushes 2023-02-12 15:00:32,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e5e89645c25979bd82f9e1e324b861b7, disabling compactions & flushes 2023-02-12 15:00:32,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:32,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:32,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:32,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:32,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. after waiting 0 ms 2023-02-12 15:00:32,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. after waiting 0 ms 2023-02-12 15:00:32,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:32,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:32,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 15:00:32,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:32,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c54693fd62fafde40475f3c394bfbc2d: Waiting for close lock at 1676214032521Running coprocessor pre-close hooks at 1676214032521Disabling compacts and flushes for region at 1676214032521Disabling writes for close at 1676214032521Writing region close event to WAL at 1676214032521Running coprocessor post-close hooks at 1676214032525 (+4 ms)Closed at 1676214032525 2023-02-12 15:00:32,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 15:00:32,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 15:00:32,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:32,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1886cfbba21a42a84ae46d52d88f5150: Waiting for close lock at 1676214032522Running coprocessor pre-close hooks at 1676214032522Disabling compacts and flushes for region at 1676214032522Disabling writes for close at 1676214032522Writing region close event to WAL at 1676214032522Running coprocessor post-close hooks at 1676214032526 (+4 ms)Closed at 1676214032526 2023-02-12 15:00:32,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:32,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e5e89645c25979bd82f9e1e324b861b7: Waiting for close lock at 1676214032522Running coprocessor pre-close hooks at 1676214032522Disabling compacts and flushes for region at 1676214032522Disabling writes for close at 1676214032522Writing region close event to WAL at 1676214032522Running coprocessor post-close hooks at 1676214032526 (+4 ms)Closed at 1676214032526 2023-02-12 15:00:32,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:32,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:32,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 629e2ec9e8223cc0fb7f0875b15cef8c, disabling compactions & flushes 2023-02-12 15:00:32,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:32,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:32,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. after waiting 0 ms 2023-02-12 15:00:32,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:32,527 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=c54693fd62fafde40475f3c394bfbc2d, regionState=CLOSED 2023-02-12 15:00:32,527 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214032527"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214032527"}]},"ts":"1676214032527"} 2023-02-12 15:00:32,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:32,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:32,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 6caf9978ca0972f75c0854f1c9fb76e3, disabling compactions & flushes 2023-02-12 15:00:32,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:32,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:32,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. after waiting 0 ms 2023-02-12 15:00:32,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:32,528 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=1886cfbba21a42a84ae46d52d88f5150, regionState=CLOSED 2023-02-12 15:00:32,528 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214032528"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214032528"}]},"ts":"1676214032528"} 2023-02-12 15:00:32,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:32,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:32,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7736b0c91a68e683d42e81f85bf8f471, disabling compactions & flushes 2023-02-12 15:00:32,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:32,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:32,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. after waiting 0 ms 2023-02-12 15:00:32,529 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:32,530 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=e5e89645c25979bd82f9e1e324b861b7, regionState=CLOSED 2023-02-12 15:00:32,530 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214032530"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214032530"}]},"ts":"1676214032530"} 2023-02-12 15:00:32,531 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=219 2023-02-12 15:00:32,531 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=219, state=SUCCESS; CloseRegionProcedure c54693fd62fafde40475f3c394bfbc2d, server=jenkins-hbase4.apache.org,37113,1676214004377 in 160 msec 2023-02-12 15:00:32,532 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=217 2023-02-12 15:00:32,532 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=217, state=SUCCESS; CloseRegionProcedure 1886cfbba21a42a84ae46d52d88f5150, server=jenkins-hbase4.apache.org,38453,1676214004461 in 159 msec 2023-02-12 15:00:32,533 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, UNASSIGN in 167 msec 2023-02-12 15:00:32,533 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=216 2023-02-12 15:00:32,533 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, UNASSIGN in 168 msec 2023-02-12 15:00:32,533 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=216, state=SUCCESS; CloseRegionProcedure e5e89645c25979bd82f9e1e324b861b7, server=jenkins-hbase4.apache.org,33845,1676214004423 in 162 msec 2023-02-12 15:00:32,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 15:00:32,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 15:00:32,534 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, UNASSIGN in 169 msec 2023-02-12 15:00:32,535 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:32,535 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 6caf9978ca0972f75c0854f1c9fb76e3: Waiting for close lock at 1676214032527Running coprocessor pre-close hooks at 1676214032527Disabling compacts and flushes for region at 1676214032527Disabling writes for close at 1676214032527Writing region close event to WAL at 1676214032528 (+1 ms)Running coprocessor post-close hooks at 1676214032535 (+7 ms)Closed at 1676214032535 2023-02-12 15:00:32,535 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:32,535 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7736b0c91a68e683d42e81f85bf8f471: Waiting for close lock at 1676214032529Running coprocessor pre-close hooks at 1676214032529Disabling compacts and flushes for region at 1676214032529Disabling writes for close at 1676214032529Writing region close event to WAL at 1676214032529Running coprocessor post-close hooks at 1676214032535 (+6 ms)Closed at 1676214032535 2023-02-12 15:00:32,535 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-12 15:00:32,535 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:32,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 629e2ec9e8223cc0fb7f0875b15cef8c: Waiting for close lock at 1676214032526Running coprocessor pre-close hooks at 1676214032526Disabling compacts and flushes for region at 1676214032526Disabling writes for close at 1676214032526Writing region close event to WAL at 1676214032527 (+1 ms)Running coprocessor post-close hooks at 1676214032535 (+8 ms)Closed at 1676214032535 2023-02-12 15:00:32,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:32,536 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=6caf9978ca0972f75c0854f1c9fb76e3, regionState=CLOSED 2023-02-12 15:00:32,536 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214032536"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214032536"}]},"ts":"1676214032536"} 2023-02-12 15:00:32,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:32,537 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=7736b0c91a68e683d42e81f85bf8f471, regionState=CLOSED 2023-02-12 15:00:32,537 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214032537"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214032537"}]},"ts":"1676214032537"} 2023-02-12 15:00:32,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:32,537 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=629e2ec9e8223cc0fb7f0875b15cef8c, regionState=CLOSED 2023-02-12 15:00:32,538 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214032537"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214032537"}]},"ts":"1676214032537"} 2023-02-12 15:00:32,539 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=221 2023-02-12 15:00:32,539 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=221, state=SUCCESS; CloseRegionProcedure 6caf9978ca0972f75c0854f1c9fb76e3, server=jenkins-hbase4.apache.org,38453,1676214004461 in 168 msec 2023-02-12 15:00:32,540 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=220 2023-02-12 15:00:32,540 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=220, state=SUCCESS; CloseRegionProcedure 7736b0c91a68e683d42e81f85bf8f471, server=jenkins-hbase4.apache.org,33845,1676214004423 in 164 msec 2023-02-12 15:00:32,541 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, UNASSIGN in 175 msec 2023-02-12 15:00:32,541 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=218 2023-02-12 15:00:32,541 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=218, state=SUCCESS; CloseRegionProcedure 629e2ec9e8223cc0fb7f0875b15cef8c, server=jenkins-hbase4.apache.org,37113,1676214004377 in 167 msec 2023-02-12 15:00:32,542 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, UNASSIGN in 176 msec 2023-02-12 15:00:32,543 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=218, resume processing ppid=215 2023-02-12 15:00:32,543 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, UNASSIGN in 177 msec 2023-02-12 15:00:32,543 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214032543"}]},"ts":"1676214032543"} 2023-02-12 15:00:32,544 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676214031582, state=DISABLED in hbase:meta 2023-02-12 15:00:32,546 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676214031582 to state=DISABLED 2023-02-12 15:00:32,548 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676214031582 in 190 msec 2023-02-12 15:00:32,571 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-12 15:00:32,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-12 15:00:32,612 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676214031582, procId: 215 completed 2023-02-12 15:00:32,613 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676214031582 table=testCorruptedSnapshot-1676214031582 type=FLUSH ttl=0 } 2023-02-12 15:00:32,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676214032613 (current time:1676214032613). 2023-02-12 15:00:32,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 15:00:32,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(139): Connect 0x6f64085f to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:32,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e8636b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:32,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:32,622 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53058, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:32,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f64085f to 127.0.0.1:61893 2023-02-12 15:00:32,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:32,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 15:00:32,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676214031582 table=testCorruptedSnapshot-1676214031582 type=FLUSH ttl=0 } 2023-02-12 15:00:32,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676214031582, type=EXCLUSIVE 2023-02-12 15:00:32,627 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676214031582, type=EXCLUSIVE 2023-02-12 15:00:32,627 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676214031582, type=EXCLUSIVE; timeout=600000, timestamp=1676214632627 2023-02-12 15:00:32,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676214031582 table=testCorruptedSnapshot-1676214031582 type=FLUSH ttl=0 } 2023-02-12 15:00:32,627 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676214031582 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676214031582 2023-02-12 15:00:32,628 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676214031582 table=testCorruptedSnapshot-1676214031582 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 15:00:32,628 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 15:00:32,643 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676214031582 table=testCorruptedSnapshot-1676214031582 type=DISABLED ttl=0 } 2023-02-12 15:00:32,645 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:32,645 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:32,645 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:32,646 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:32,648 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:32,648 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:32,648 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:32,648 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:32,649 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:32,649 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:32,649 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:32,649 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:32,649 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:32,649 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:32,650 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:32,650 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:32,650 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:32,651 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:32,651 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:32,652 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:32,655 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-12 15:00:32,727 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676214031582 table=testCorruptedSnapshot-1676214031582 type=DISABLED ttl=0 } as finished. 2023-02-12 15:00:32,727 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676214031582 2023-02-12 15:00:32,727 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/emptySnaptb-1676214031582 2023-02-12 15:00:32,755 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/emptySnaptb-1676214031582 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/emptySnaptb-1676214031582 2023-02-12 15:00:32,779 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676214031582 of table testCorruptedSnapshot-1676214031582 completed 2023-02-12 15:00:32,779 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/emptySnaptb-1676214031582 2023-02-12 15:00:32,780 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/emptySnaptb-1676214031582 2023-02-12 15:00:32,780 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676214031582 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676214031582 at 1676214032627Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676214031582 table=testCorruptedSnapshot-1676214031582 type=DISABLED ttl=0 } at 1676214032644 (+17 ms)Consolidate snapshot: emptySnaptb-1676214031582 at 1676214032727 (+83 ms)Loading Region manifests for emptySnaptb-1676214031582 at 1676214032727Writing data manifest for emptySnaptb-1676214031582 at 1676214032732 (+5 ms)Verifying snapshot: emptySnaptb-1676214031582 at 1676214032744 (+12 ms)Snapshot emptySnaptb-1676214031582 of table testCorruptedSnapshot-1676214031582 completed at 1676214032779 (+35 ms) 2023-02-12 15:00:32,781 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676214031582, type=EXCLUSIVE 2023-02-12 15:00:32,782 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676214031582, type=EXCLUSIVE in 156 msec 2023-02-12 15:00:32,878 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 15:00:32,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676214031582 table=testCorruptedSnapshot-1676214031582 type=FLUSH ttl=0 } is done 2023-02-12 15:00:32,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676214031582 table=testCorruptedSnapshot-1676214031582 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 15:00:32,879 INFO [Listener at localhost/38323] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676214031582 2023-02-12 15:00:32,880 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676214031582 2023-02-12 15:00:32,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676214031582 2023-02-12 15:00:32,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-12 15:00:32,883 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676214031582 2023-02-12 15:00:32,883 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214032883"}]},"ts":"1676214032883"} 2023-02-12 15:00:32,884 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676214031582, state=ENABLING in hbase:meta 2023-02-12 15:00:32,886 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 15:00:32,886 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, ASSIGN}] 2023-02-12 15:00:32,888 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, ASSIGN 2023-02-12 15:00:32,888 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, ASSIGN 2023-02-12 15:00:32,888 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, ASSIGN 2023-02-12 15:00:32,889 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, ASSIGN 2023-02-12 15:00:32,889 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, ASSIGN 2023-02-12 15:00:32,889 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:32,889 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:32,889 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=true 2023-02-12 15:00:32,889 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=true 2023-02-12 15:00:32,889 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:32,889 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:32,889 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=true 2023-02-12 15:00:32,889 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:32,889 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=true 2023-02-12 15:00:32,890 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=true 2023-02-12 15:00:32,890 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, ASSIGN 2023-02-12 15:00:32,891 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:32,891 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=true 2023-02-12 15:00:33,039 INFO [jenkins-hbase4:45369] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 15:00:33,040 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=1886cfbba21a42a84ae46d52d88f5150, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:33,040 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=7736b0c91a68e683d42e81f85bf8f471, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:33,040 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=629e2ec9e8223cc0fb7f0875b15cef8c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:33,040 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214033040"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214033040"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214033040"}]},"ts":"1676214033040"} 2023-02-12 15:00:33,040 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214033040"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214033040"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214033040"}]},"ts":"1676214033040"} 2023-02-12 15:00:33,040 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=c54693fd62fafde40475f3c394bfbc2d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:33,040 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=6caf9978ca0972f75c0854f1c9fb76e3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:33,040 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214033040"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214033040"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214033040"}]},"ts":"1676214033040"} 2023-02-12 15:00:33,040 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214033040"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214033040"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214033040"}]},"ts":"1676214033040"} 2023-02-12 15:00:33,040 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214033040"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214033040"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214033040"}]},"ts":"1676214033040"} 2023-02-12 15:00:33,041 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=231, state=RUNNABLE; OpenRegionProcedure 1886cfbba21a42a84ae46d52d88f5150, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:33,042 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=232, state=RUNNABLE; OpenRegionProcedure 629e2ec9e8223cc0fb7f0875b15cef8c, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:33,043 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=233, state=RUNNABLE; OpenRegionProcedure c54693fd62fafde40475f3c394bfbc2d, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:33,044 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=234, state=RUNNABLE; OpenRegionProcedure 7736b0c91a68e683d42e81f85bf8f471, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:33,044 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=235, state=RUNNABLE; OpenRegionProcedure 6caf9978ca0972f75c0854f1c9fb76e3, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:33,047 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=e5e89645c25979bd82f9e1e324b861b7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:33,047 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214033047"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214033047"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214033047"}]},"ts":"1676214033047"} 2023-02-12 15:00:33,048 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=230, state=RUNNABLE; OpenRegionProcedure e5e89645c25979bd82f9e1e324b861b7, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:33,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-12 15:00:33,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:33,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:33,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c54693fd62fafde40475f3c394bfbc2d, NAME => 'testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 15:00:33,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1886cfbba21a42a84ae46d52d88f5150, NAME => 'testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 15:00:33,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:33,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:33,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:33,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:33,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:33,196 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:33,197 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:33,197 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:33,198 INFO [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:33,198 INFO [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:33,198 DEBUG [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf 2023-02-12 15:00:33,198 DEBUG [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf 2023-02-12 15:00:33,198 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:33,198 DEBUG [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf 2023-02-12 15:00:33,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e5e89645c25979bd82f9e1e324b861b7, NAME => 'testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 15:00:33,199 DEBUG [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf 2023-02-12 15:00:33,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:33,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:33,199 INFO [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c54693fd62fafde40475f3c394bfbc2d columnFamilyName cf 2023-02-12 15:00:33,199 INFO [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1886cfbba21a42a84ae46d52d88f5150 columnFamilyName cf 2023-02-12 15:00:33,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:33,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:33,200 INFO [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] regionserver.HStore(310): Store=c54693fd62fafde40475f3c394bfbc2d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:33,200 INFO [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] regionserver.HStore(310): Store=1886cfbba21a42a84ae46d52d88f5150/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:33,200 INFO [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:33,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:33,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:33,201 DEBUG [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf 2023-02-12 15:00:33,201 DEBUG [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf 2023-02-12 15:00:33,201 INFO [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e5e89645c25979bd82f9e1e324b861b7 columnFamilyName cf 2023-02-12 15:00:33,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:33,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:33,202 INFO [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] regionserver.HStore(310): Store=e5e89645c25979bd82f9e1e324b861b7/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:33,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:33,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:33,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:33,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:33,205 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1886cfbba21a42a84ae46d52d88f5150; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73159922, jitterRate=0.09016779065132141}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:33,205 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c54693fd62fafde40475f3c394bfbc2d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75203738, jitterRate=0.12062302231788635}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:33,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1886cfbba21a42a84ae46d52d88f5150: Running coprocessor pre-open hook at 1676214033197Writing region info on filesystem at 1676214033197Initializing all the Stores at 1676214033197Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214033197Cleaning up temporary data from old regions at 1676214033202 (+5 ms)Cleaning up detritus from prior splits at 1676214033202Running coprocessor post-open hooks at 1676214033205 (+3 ms)Region opened successfully at 1676214033205 2023-02-12 15:00:33,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c54693fd62fafde40475f3c394bfbc2d: Running coprocessor pre-open hook at 1676214033197Writing region info on filesystem at 1676214033197Initializing all the Stores at 1676214033197Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214033197Cleaning up temporary data from old regions at 1676214033202 (+5 ms)Cleaning up detritus from prior splits at 1676214033202Running coprocessor post-open hooks at 1676214033205 (+3 ms)Region opened successfully at 1676214033205 2023-02-12 15:00:33,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:33,205 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d., pid=238, masterSystemTime=1676214033194 2023-02-12 15:00:33,205 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150., pid=236, masterSystemTime=1676214033194 2023-02-12 15:00:33,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e5e89645c25979bd82f9e1e324b861b7; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72653479, jitterRate=0.08262120187282562}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:33,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e5e89645c25979bd82f9e1e324b861b7: Running coprocessor pre-open hook at 1676214033199Writing region info on filesystem at 1676214033199Initializing all the Stores at 1676214033200 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214033200Cleaning up temporary data from old regions at 1676214033203 (+3 ms)Cleaning up detritus from prior splits at 1676214033204 (+1 ms)Running coprocessor post-open hooks at 1676214033206 (+2 ms)Region opened successfully at 1676214033206 2023-02-12 15:00:33,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:33,207 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:33,207 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:33,207 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7., pid=241, masterSystemTime=1676214033197 2023-02-12 15:00:33,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 629e2ec9e8223cc0fb7f0875b15cef8c, NAME => 'testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 15:00:33,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:33,207 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=c54693fd62fafde40475f3c394bfbc2d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:33,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:33,207 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:33,207 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214033207"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214033207"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214033207"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214033207"}]},"ts":"1676214033207"} 2023-02-12 15:00:33,207 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:33,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:33,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6caf9978ca0972f75c0854f1c9fb76e3, NAME => 'testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 15:00:33,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:33,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:33,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:33,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:33,208 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=1886cfbba21a42a84ae46d52d88f5150, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:33,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:33,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:33,208 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214033208"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214033208"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214033208"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214033208"}]},"ts":"1676214033208"} 2023-02-12 15:00:33,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:33,208 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:33,208 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:33,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7736b0c91a68e683d42e81f85bf8f471, NAME => 'testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 15:00:33,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:33,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:33,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:33,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:33,209 INFO [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:33,209 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=e5e89645c25979bd82f9e1e324b861b7, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:33,209 INFO [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:33,209 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214033209"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214033209"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214033209"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214033209"}]},"ts":"1676214033209"} 2023-02-12 15:00:33,210 INFO [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:33,210 DEBUG [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf 2023-02-12 15:00:33,210 DEBUG [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf 2023-02-12 15:00:33,210 DEBUG [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf 2023-02-12 15:00:33,210 DEBUG [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf 2023-02-12 15:00:33,211 INFO [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 629e2ec9e8223cc0fb7f0875b15cef8c columnFamilyName cf 2023-02-12 15:00:33,211 INFO [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6caf9978ca0972f75c0854f1c9fb76e3 columnFamilyName cf 2023-02-12 15:00:33,211 DEBUG [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf 2023-02-12 15:00:33,211 DEBUG [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf 2023-02-12 15:00:33,212 INFO [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7736b0c91a68e683d42e81f85bf8f471 columnFamilyName cf 2023-02-12 15:00:33,212 INFO [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] regionserver.HStore(310): Store=629e2ec9e8223cc0fb7f0875b15cef8c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:33,212 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=233 2023-02-12 15:00:33,212 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=233, state=SUCCESS; OpenRegionProcedure c54693fd62fafde40475f3c394bfbc2d, server=jenkins-hbase4.apache.org,37113,1676214004377 in 166 msec 2023-02-12 15:00:33,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:33,213 INFO [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] regionserver.HStore(310): Store=6caf9978ca0972f75c0854f1c9fb76e3/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:33,213 INFO [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] regionserver.HStore(310): Store=7736b0c91a68e683d42e81f85bf8f471/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:33,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:33,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:33,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:33,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:33,215 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=230 2023-02-12 15:00:33,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:33,215 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=231 2023-02-12 15:00:33,215 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, ASSIGN in 326 msec 2023-02-12 15:00:33,215 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=231, state=SUCCESS; OpenRegionProcedure 1886cfbba21a42a84ae46d52d88f5150, server=jenkins-hbase4.apache.org,38453,1676214004461 in 170 msec 2023-02-12 15:00:33,215 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=230, state=SUCCESS; OpenRegionProcedure e5e89645c25979bd82f9e1e324b861b7, server=jenkins-hbase4.apache.org,33845,1676214004423 in 164 msec 2023-02-12 15:00:33,216 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, ASSIGN in 329 msec 2023-02-12 15:00:33,216 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, ASSIGN in 329 msec 2023-02-12 15:00:33,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:33,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:33,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 629e2ec9e8223cc0fb7f0875b15cef8c; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71223227, jitterRate=0.061308786273002625}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:33,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 629e2ec9e8223cc0fb7f0875b15cef8c: Running coprocessor pre-open hook at 1676214033208Writing region info on filesystem at 1676214033208Initializing all the Stores at 1676214033209 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214033209Cleaning up temporary data from old regions at 1676214033214 (+5 ms)Cleaning up detritus from prior splits at 1676214033215 (+1 ms)Running coprocessor post-open hooks at 1676214033217 (+2 ms)Region opened successfully at 1676214033217 2023-02-12 15:00:33,218 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 6caf9978ca0972f75c0854f1c9fb76e3; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59012298, jitterRate=-0.12064823508262634}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:33,218 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c., pid=237, masterSystemTime=1676214033194 2023-02-12 15:00:33,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 6caf9978ca0972f75c0854f1c9fb76e3: Running coprocessor pre-open hook at 1676214033208Writing region info on filesystem at 1676214033208Initializing all the Stores at 1676214033209 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214033209Cleaning up temporary data from old regions at 1676214033215 (+6 ms)Cleaning up detritus from prior splits at 1676214033215Running coprocessor post-open hooks at 1676214033218 (+3 ms)Region opened successfully at 1676214033218 2023-02-12 15:00:33,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3., pid=240, masterSystemTime=1676214033194 2023-02-12 15:00:33,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:33,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:33,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:33,220 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7736b0c91a68e683d42e81f85bf8f471; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74208017, jitterRate=0.10578562319278717}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:33,220 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=629e2ec9e8223cc0fb7f0875b15cef8c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:33,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:33,220 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214033219"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214033219"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214033219"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214033219"}]},"ts":"1676214033219"} 2023-02-12 15:00:33,220 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:33,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7736b0c91a68e683d42e81f85bf8f471: Running coprocessor pre-open hook at 1676214033209Writing region info on filesystem at 1676214033209Initializing all the Stores at 1676214033210 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214033210Cleaning up temporary data from old regions at 1676214033215 (+5 ms)Cleaning up detritus from prior splits at 1676214033216 (+1 ms)Running coprocessor post-open hooks at 1676214033220 (+4 ms)Region opened successfully at 1676214033220 2023-02-12 15:00:33,220 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=6caf9978ca0972f75c0854f1c9fb76e3, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:33,220 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214033220"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214033220"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214033220"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214033220"}]},"ts":"1676214033220"} 2023-02-12 15:00:33,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471., pid=239, masterSystemTime=1676214033197 2023-02-12 15:00:33,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:33,222 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:33,222 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=7736b0c91a68e683d42e81f85bf8f471, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:33,222 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214033222"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214033222"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214033222"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214033222"}]},"ts":"1676214033222"} 2023-02-12 15:00:33,223 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=232 2023-02-12 15:00:33,223 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=232, state=SUCCESS; OpenRegionProcedure 629e2ec9e8223cc0fb7f0875b15cef8c, server=jenkins-hbase4.apache.org,37113,1676214004377 in 179 msec 2023-02-12 15:00:33,225 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, ASSIGN in 337 msec 2023-02-12 15:00:33,225 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=235 2023-02-12 15:00:33,225 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=235, state=SUCCESS; OpenRegionProcedure 6caf9978ca0972f75c0854f1c9fb76e3, server=jenkins-hbase4.apache.org,38453,1676214004461 in 178 msec 2023-02-12 15:00:33,226 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, ASSIGN in 339 msec 2023-02-12 15:00:33,226 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=234 2023-02-12 15:00:33,226 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=234, state=SUCCESS; OpenRegionProcedure 7736b0c91a68e683d42e81f85bf8f471, server=jenkins-hbase4.apache.org,33845,1676214004423 in 181 msec 2023-02-12 15:00:33,227 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=234, resume processing ppid=229 2023-02-12 15:00:33,227 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, ASSIGN in 340 msec 2023-02-12 15:00:33,228 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214033228"}]},"ts":"1676214033228"} 2023-02-12 15:00:33,229 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676214031582, state=ENABLED in hbase:meta 2023-02-12 15:00:33,234 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676214031582' was successfully enabled. 2023-02-12 15:00:33,236 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676214031582 in 355 msec 2023-02-12 15:00:33,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-12 15:00:33,634 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676214031582, procId: 229 completed 2023-02-12 15:00:33,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:33,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:33,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:33,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:33,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:33,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:33,648 DEBUG [Listener at localhost/38323] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676214031582 2023-02-12 15:00:33,648 DEBUG [Listener at localhost/38323] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:33,648 INFO [Listener at localhost/38323] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-12 15:00:33,660 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 15:00:33,661 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 15:00:33,662 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 15:00:33,662 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 15:00:33,664 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 15:00:33,693 INFO [Listener at localhost/38323] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676214031582 2023-02-12 15:00:33,693 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ff51970 to 127.0.0.1:61893 2023-02-12 15:00:33,693 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:33,693 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676214031582 2023-02-12 15:00:33,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676214031582 2023-02-12 15:00:33,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-12 15:00:33,698 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214033698"}]},"ts":"1676214033698"} 2023-02-12 15:00:33,699 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676214031582, state=DISABLING in hbase:meta 2023-02-12 15:00:33,799 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676214031582 to state=DISABLING 2023-02-12 15:00:33,800 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, UNASSIGN}] 2023-02-12 15:00:33,802 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, UNASSIGN 2023-02-12 15:00:33,802 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, UNASSIGN 2023-02-12 15:00:33,802 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, UNASSIGN 2023-02-12 15:00:33,802 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, UNASSIGN 2023-02-12 15:00:33,802 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, UNASSIGN 2023-02-12 15:00:33,803 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=7736b0c91a68e683d42e81f85bf8f471, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:33,803 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=629e2ec9e8223cc0fb7f0875b15cef8c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:33,803 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214033803"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214033803"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214033803"}]},"ts":"1676214033803"} 2023-02-12 15:00:33,803 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214033803"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214033803"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214033803"}]},"ts":"1676214033803"} 2023-02-12 15:00:33,803 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=1886cfbba21a42a84ae46d52d88f5150, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:33,803 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=c54693fd62fafde40475f3c394bfbc2d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:33,803 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214033803"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214033803"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214033803"}]},"ts":"1676214033803"} 2023-02-12 15:00:33,803 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214033803"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214033803"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214033803"}]},"ts":"1676214033803"} 2023-02-12 15:00:33,803 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=6caf9978ca0972f75c0854f1c9fb76e3, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:33,804 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214033803"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214033803"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214033803"}]},"ts":"1676214033803"} 2023-02-12 15:00:33,804 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=247, state=RUNNABLE; CloseRegionProcedure 7736b0c91a68e683d42e81f85bf8f471, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:33,805 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=245, state=RUNNABLE; CloseRegionProcedure 629e2ec9e8223cc0fb7f0875b15cef8c, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:33,806 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=244, state=RUNNABLE; CloseRegionProcedure 1886cfbba21a42a84ae46d52d88f5150, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:33,807 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=246, state=RUNNABLE; CloseRegionProcedure c54693fd62fafde40475f3c394bfbc2d, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:33,808 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, UNASSIGN 2023-02-12 15:00:33,808 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=248, state=RUNNABLE; CloseRegionProcedure 6caf9978ca0972f75c0854f1c9fb76e3, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:33,808 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=e5e89645c25979bd82f9e1e324b861b7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:33,808 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214033808"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214033808"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214033808"}]},"ts":"1676214033808"} 2023-02-12 15:00:33,810 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=243, state=RUNNABLE; CloseRegionProcedure e5e89645c25979bd82f9e1e324b861b7, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:33,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-12 15:00:33,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:33,957 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e5e89645c25979bd82f9e1e324b861b7, disabling compactions & flushes 2023-02-12 15:00:33,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:33,957 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:33,957 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. after waiting 0 ms 2023-02-12 15:00:33,957 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:33,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e5e89645c25979bd82f9e1e324b861b7 1/1 column families, dataSize=2.48 KB heapSize=5.59 KB 2023-02-12 15:00:33,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:33,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:33,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 629e2ec9e8223cc0fb7f0875b15cef8c, disabling compactions & flushes 2023-02-12 15:00:33,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 6caf9978ca0972f75c0854f1c9fb76e3, disabling compactions & flushes 2023-02-12 15:00:33,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:33,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:33,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:33,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. after waiting 0 ms 2023-02-12 15:00:33,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:33,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:33,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. after waiting 0 ms 2023-02-12 15:00:33,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 629e2ec9e8223cc0fb7f0875b15cef8c 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-12 15:00:33,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:33,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 6caf9978ca0972f75c0854f1c9fb76e3 1/1 column families, dataSize=22.37 KB heapSize=48.34 KB 2023-02-12 15:00:33,991 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302124cf05442fae6401aa73faf7b5237b337 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c81e728d9d4c2f636f067f89cc14862c202302124cf05442fae6401aa73faf7b5237b337 2023-02-12 15:00:33,991 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230212c91709444cd846a4bf112a5d5151b599 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/d41d8cd98f00b204e9800998ecf8427e20230212c91709444cd846a4bf112a5d5151b599 2023-02-12 15:00:33,993 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302120e96bf35121a4a4baf068cc3019938dc to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/e4da3b7fbbce2345d7772b0674a318d5202302120e96bf35121a4a4baf068cc3019938dc 2023-02-12 15:00:33,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/.tmp/cf/7f393b696f8949d1994c4ab9ad8fa066 2023-02-12 15:00:33,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/.tmp/cf/03355b39c2174ea5b1a94a16f11cc6f5 2023-02-12 15:00:34,001 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/.tmp/cf/8548c4a9d6ea4c02a2357194a5e51e1c 2023-02-12 15:00:34,003 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/.tmp/cf/03355b39c2174ea5b1a94a16f11cc6f5 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/03355b39c2174ea5b1a94a16f11cc6f5 2023-02-12 15:00:34,003 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/.tmp/cf/7f393b696f8949d1994c4ab9ad8fa066 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf/7f393b696f8949d1994c4ab9ad8fa066 2023-02-12 15:00:34,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/.tmp/cf/8548c4a9d6ea4c02a2357194a5e51e1c as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf/8548c4a9d6ea4c02a2357194a5e51e1c 2023-02-12 15:00:34,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/03355b39c2174ea5b1a94a16f11cc6f5, entries=34, sequenceid=8, filesize=10.8 K 2023-02-12 15:00:34,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 629e2ec9e8223cc0fb7f0875b15cef8c in 48ms, sequenceid=8, compaction requested=false 2023-02-12 15:00:34,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676214031582' 2023-02-12 15:00:34,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf/8548c4a9d6ea4c02a2357194a5e51e1c, entries=342, sequenceid=8, filesize=63.8 K 2023-02-12 15:00:34,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.37 KB/22910, heapSize ~48.33 KB/49488, currentSize=0 B/0 for 6caf9978ca0972f75c0854f1c9fb76e3 in 49ms, sequenceid=8, compaction requested=false 2023-02-12 15:00:34,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf/7f393b696f8949d1994c4ab9ad8fa066, entries=38, sequenceid=8, filesize=11.4 K 2023-02-12 15:00:34,011 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for e5e89645c25979bd82f9e1e324b861b7 in 54ms, sequenceid=8, compaction requested=false 2023-02-12 15:00:34,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 15:00:34,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:34,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 629e2ec9e8223cc0fb7f0875b15cef8c: Waiting for close lock at 1676214033959Running coprocessor pre-close hooks at 1676214033959Disabling compacts and flushes for region at 1676214033959Disabling writes for close at 1676214033959Obtaining lock to block concurrent updates at 1676214033959Preparing flush snapshotting stores in 629e2ec9e8223cc0fb7f0875b15cef8c at 1676214033959Finished memstore snapshotting testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c., syncing WAL and waiting on mvcc, flushsize=dataSize=2276, getHeapSize=5136, getOffHeapSize=0, getCellsCount=34 at 1676214033960 (+1 ms)Flushing stores of testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. at 1676214033960Flushing 629e2ec9e8223cc0fb7f0875b15cef8c/cf: creating writer at 1676214033961 (+1 ms)Flushing mob file 629e2ec9e8223cc0fb7f0875b15cef8c/cf: appending metadata at 1676214033965 (+4 ms)Flushing mob file 629e2ec9e8223cc0fb7f0875b15cef8c/cf: closing flushed file at 1676214033965Flushing 629e2ec9e8223cc0fb7f0875b15cef8c/cf: appending metadata at 1676214033992 (+27 ms)Flushing 629e2ec9e8223cc0fb7f0875b15cef8c/cf: closing flushed file at 1676214033992Flushing 629e2ec9e8223cc0fb7f0875b15cef8c/cf: reopening flushed file at 1676214034003 (+11 ms)Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 629e2ec9e8223cc0fb7f0875b15cef8c in 48ms, sequenceid=8, compaction requested=false at 1676214034007 (+4 ms)Writing region close event to WAL at 1676214034008 (+1 ms)Running coprocessor post-close hooks at 1676214034017 (+9 ms)Closed at 1676214034017 2023-02-12 15:00:34,019 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=629e2ec9e8223cc0fb7f0875b15cef8c, regionState=CLOSED 2023-02-12 15:00:34,019 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214034019"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214034019"}]},"ts":"1676214034019"} 2023-02-12 15:00:34,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 15:00:34,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:34,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 15:00:34,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:34,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c54693fd62fafde40475f3c394bfbc2d, disabling compactions & flushes 2023-02-12 15:00:34,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:34,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:34,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. after waiting 0 ms 2023-02-12 15:00:34,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:34,022 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c54693fd62fafde40475f3c394bfbc2d 1/1 column families, dataSize=1.11 KB heapSize=2.64 KB 2023-02-12 15:00:34,022 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=245 2023-02-12 15:00:34,022 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:34,023 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=245, state=SUCCESS; CloseRegionProcedure 629e2ec9e8223cc0fb7f0875b15cef8c, server=jenkins-hbase4.apache.org,37113,1676214004377 in 215 msec 2023-02-12 15:00:34,023 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e5e89645c25979bd82f9e1e324b861b7: Waiting for close lock at 1676214033957Running coprocessor pre-close hooks at 1676214033957Disabling compacts and flushes for region at 1676214033957Disabling writes for close at 1676214033957Obtaining lock to block concurrent updates at 1676214033957Preparing flush snapshotting stores in e5e89645c25979bd82f9e1e324b861b7 at 1676214033957Finished memstore snapshotting testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7., syncing WAL and waiting on mvcc, flushsize=dataSize=2544, getHeapSize=5712, getOffHeapSize=0, getCellsCount=38 at 1676214033957Flushing stores of testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. at 1676214033958 (+1 ms)Flushing e5e89645c25979bd82f9e1e324b861b7/cf: creating writer at 1676214033959 (+1 ms)Flushing mob file e5e89645c25979bd82f9e1e324b861b7/cf: appending metadata at 1676214033965 (+6 ms)Flushing mob file e5e89645c25979bd82f9e1e324b861b7/cf: closing flushed file at 1676214033965Flushing e5e89645c25979bd82f9e1e324b861b7/cf: appending metadata at 1676214033992 (+27 ms)Flushing e5e89645c25979bd82f9e1e324b861b7/cf: closing flushed file at 1676214033992Flushing e5e89645c25979bd82f9e1e324b861b7/cf: reopening flushed file at 1676214034004 (+12 ms)Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for e5e89645c25979bd82f9e1e324b861b7 in 54ms, sequenceid=8, compaction requested=false at 1676214034011 (+7 ms)Writing region close event to WAL at 1676214034016 (+5 ms)Running coprocessor post-close hooks at 1676214034022 (+6 ms)Closed at 1676214034022 2023-02-12 15:00:34,023 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:34,023 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 6caf9978ca0972f75c0854f1c9fb76e3: Waiting for close lock at 1676214033959Running coprocessor pre-close hooks at 1676214033959Disabling compacts and flushes for region at 1676214033959Disabling writes for close at 1676214033959Obtaining lock to block concurrent updates at 1676214033960 (+1 ms)Preparing flush snapshotting stores in 6caf9978ca0972f75c0854f1c9fb76e3 at 1676214033960Finished memstore snapshotting testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3., syncing WAL and waiting on mvcc, flushsize=dataSize=22910, getHeapSize=49488, getOffHeapSize=0, getCellsCount=342 at 1676214033960Flushing stores of testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. at 1676214033961 (+1 ms)Flushing 6caf9978ca0972f75c0854f1c9fb76e3/cf: creating writer at 1676214033962 (+1 ms)Flushing mob file 6caf9978ca0972f75c0854f1c9fb76e3/cf: appending metadata at 1676214033967 (+5 ms)Flushing mob file 6caf9978ca0972f75c0854f1c9fb76e3/cf: closing flushed file at 1676214033967Flushing 6caf9978ca0972f75c0854f1c9fb76e3/cf: appending metadata at 1676214033994 (+27 ms)Flushing 6caf9978ca0972f75c0854f1c9fb76e3/cf: closing flushed file at 1676214033994Flushing 6caf9978ca0972f75c0854f1c9fb76e3/cf: reopening flushed file at 1676214034005 (+11 ms)Finished flush of dataSize ~22.37 KB/22910, heapSize ~48.33 KB/49488, currentSize=0 B/0 for 6caf9978ca0972f75c0854f1c9fb76e3 in 49ms, sequenceid=8, compaction requested=false at 1676214034008 (+3 ms)Writing region close event to WAL at 1676214034016 (+8 ms)Running coprocessor post-close hooks at 1676214034023 (+7 ms)Closed at 1676214034023 2023-02-12 15:00:34,024 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, UNASSIGN in 223 msec 2023-02-12 15:00:34,024 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:34,024 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:34,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1886cfbba21a42a84ae46d52d88f5150, disabling compactions & flushes 2023-02-12 15:00:34,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:34,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:34,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. after waiting 0 ms 2023-02-12 15:00:34,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:34,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1886cfbba21a42a84ae46d52d88f5150 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-12 15:00:34,025 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=6caf9978ca0972f75c0854f1c9fb76e3, regionState=CLOSED 2023-02-12 15:00:34,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:34,025 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214034025"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214034025"}]},"ts":"1676214034025"} 2023-02-12 15:00:34,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:34,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7736b0c91a68e683d42e81f85bf8f471, disabling compactions & flushes 2023-02-12 15:00:34,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:34,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:34,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. after waiting 0 ms 2023-02-12 15:00:34,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:34,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7736b0c91a68e683d42e81f85bf8f471 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-12 15:00:34,025 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=e5e89645c25979bd82f9e1e324b861b7, regionState=CLOSED 2023-02-12 15:00:34,026 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214034025"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214034025"}]},"ts":"1676214034025"} 2023-02-12 15:00:34,032 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=248 2023-02-12 15:00:34,032 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=248, state=SUCCESS; CloseRegionProcedure 6caf9978ca0972f75c0854f1c9fb76e3, server=jenkins-hbase4.apache.org,38453,1676214004461 in 221 msec 2023-02-12 15:00:34,033 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=243 2023-02-12 15:00:34,033 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=243, state=SUCCESS; CloseRegionProcedure e5e89645c25979bd82f9e1e324b861b7, server=jenkins-hbase4.apache.org,33845,1676214004423 in 220 msec 2023-02-12 15:00:34,033 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, UNASSIGN in 232 msec 2023-02-12 15:00:34,034 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, UNASSIGN in 233 msec 2023-02-12 15:00:34,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230212610045b633744904b62a524e2fd91898 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c4ca4238a0b923820dcc509a6f75849b20230212610045b633744904b62a524e2fd91898 2023-02-12 15:00:34,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230212af309e19f478444b804a53dd73a9b94b to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212af309e19f478444b804a53dd73a9b94b 2023-02-12 15:00:34,048 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c2023021241aaa89f281348dd86ac054a756c55f2 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/a87ff679a2f3e71d9181a67b7542122c2023021241aaa89f281348dd86ac054a756c55f2 2023-02-12 15:00:34,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/.tmp/cf/db6bc7be9e7747aabb3aedbe022bfbf2 2023-02-12 15:00:34,058 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/.tmp/cf/e63f254e2aa0453ea88616fb006ac4ba 2023-02-12 15:00:34,059 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/.tmp/cf/7f593598d537472990eccbd63800672b 2023-02-12 15:00:34,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/.tmp/cf/db6bc7be9e7747aabb3aedbe022bfbf2 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf/db6bc7be9e7747aabb3aedbe022bfbf2 2023-02-12 15:00:34,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/.tmp/cf/e63f254e2aa0453ea88616fb006ac4ba as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf/e63f254e2aa0453ea88616fb006ac4ba 2023-02-12 15:00:34,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/.tmp/cf/7f593598d537472990eccbd63800672b as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf/7f593598d537472990eccbd63800672b 2023-02-12 15:00:34,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf/db6bc7be9e7747aabb3aedbe022bfbf2, entries=30, sequenceid=8, filesize=10.1 K 2023-02-12 15:00:34,064 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 1886cfbba21a42a84ae46d52d88f5150 in 39ms, sequenceid=8, compaction requested=false 2023-02-12 15:00:34,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf/e63f254e2aa0453ea88616fb006ac4ba, entries=17, sequenceid=8, filesize=7.8 K 2023-02-12 15:00:34,066 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf/7f593598d537472990eccbd63800672b, entries=39, sequenceid=8, filesize=11.6 K 2023-02-12 15:00:34,066 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.11 KB/1137, heapSize ~2.63 KB/2688, currentSize=0 B/0 for c54693fd62fafde40475f3c394bfbc2d in 45ms, sequenceid=8, compaction requested=false 2023-02-12 15:00:34,066 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 7736b0c91a68e683d42e81f85bf8f471 in 41ms, sequenceid=8, compaction requested=false 2023-02-12 15:00:34,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 15:00:34,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:34,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1886cfbba21a42a84ae46d52d88f5150: Waiting for close lock at 1676214034025Running coprocessor pre-close hooks at 1676214034025Disabling compacts and flushes for region at 1676214034025Disabling writes for close at 1676214034025Obtaining lock to block concurrent updates at 1676214034025Preparing flush snapshotting stores in 1886cfbba21a42a84ae46d52d88f5150 at 1676214034025Finished memstore snapshotting testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676214034025Flushing stores of testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. at 1676214034025Flushing 1886cfbba21a42a84ae46d52d88f5150/cf: creating writer at 1676214034026 (+1 ms)Flushing mob file 1886cfbba21a42a84ae46d52d88f5150/cf: appending metadata at 1676214034036 (+10 ms)Flushing mob file 1886cfbba21a42a84ae46d52d88f5150/cf: closing flushed file at 1676214034036Flushing 1886cfbba21a42a84ae46d52d88f5150/cf: appending metadata at 1676214034047 (+11 ms)Flushing 1886cfbba21a42a84ae46d52d88f5150/cf: closing flushed file at 1676214034048 (+1 ms)Flushing 1886cfbba21a42a84ae46d52d88f5150/cf: reopening flushed file at 1676214034061 (+13 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 1886cfbba21a42a84ae46d52d88f5150 in 39ms, sequenceid=8, compaction requested=false at 1676214034064 (+3 ms)Writing region close event to WAL at 1676214034065 (+1 ms)Running coprocessor post-close hooks at 1676214034068 (+3 ms)Closed at 1676214034068 2023-02-12 15:00:34,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 15:00:34,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:34,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-12 15:00:34,069 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=1886cfbba21a42a84ae46d52d88f5150, regionState=CLOSED 2023-02-12 15:00:34,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:34,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:34,069 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214034069"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214034069"}]},"ts":"1676214034069"} 2023-02-12 15:00:34,070 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7736b0c91a68e683d42e81f85bf8f471: Waiting for close lock at 1676214034025Running coprocessor pre-close hooks at 1676214034025Disabling compacts and flushes for region at 1676214034025Disabling writes for close at 1676214034025Obtaining lock to block concurrent updates at 1676214034025Preparing flush snapshotting stores in 7736b0c91a68e683d42e81f85bf8f471 at 1676214034025Finished memstore snapshotting testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471., syncing WAL and waiting on mvcc, flushsize=dataSize=2611, getHeapSize=5856, getOffHeapSize=0, getCellsCount=39 at 1676214034026 (+1 ms)Flushing stores of testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. at 1676214034027 (+1 ms)Flushing 7736b0c91a68e683d42e81f85bf8f471/cf: creating writer at 1676214034029 (+2 ms)Flushing mob file 7736b0c91a68e683d42e81f85bf8f471/cf: appending metadata at 1676214034037 (+8 ms)Flushing mob file 7736b0c91a68e683d42e81f85bf8f471/cf: closing flushed file at 1676214034037Flushing 7736b0c91a68e683d42e81f85bf8f471/cf: appending metadata at 1676214034049 (+12 ms)Flushing 7736b0c91a68e683d42e81f85bf8f471/cf: closing flushed file at 1676214034049Flushing 7736b0c91a68e683d42e81f85bf8f471/cf: reopening flushed file at 1676214034063 (+14 ms)Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for 7736b0c91a68e683d42e81f85bf8f471 in 41ms, sequenceid=8, compaction requested=false at 1676214034066 (+3 ms)Writing region close event to WAL at 1676214034067 (+1 ms)Running coprocessor post-close hooks at 1676214034069 (+2 ms)Closed at 1676214034069 2023-02-12 15:00:34,069 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c54693fd62fafde40475f3c394bfbc2d: Waiting for close lock at 1676214034021Running coprocessor pre-close hooks at 1676214034021Disabling compacts and flushes for region at 1676214034021Disabling writes for close at 1676214034021Obtaining lock to block concurrent updates at 1676214034022 (+1 ms)Preparing flush snapshotting stores in c54693fd62fafde40475f3c394bfbc2d at 1676214034022Finished memstore snapshotting testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d., syncing WAL and waiting on mvcc, flushsize=dataSize=1137, getHeapSize=2688, getOffHeapSize=0, getCellsCount=17 at 1676214034022Flushing stores of testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. at 1676214034022Flushing c54693fd62fafde40475f3c394bfbc2d/cf: creating writer at 1676214034023 (+1 ms)Flushing mob file c54693fd62fafde40475f3c394bfbc2d/cf: appending metadata at 1676214034030 (+7 ms)Flushing mob file c54693fd62fafde40475f3c394bfbc2d/cf: closing flushed file at 1676214034030Flushing c54693fd62fafde40475f3c394bfbc2d/cf: appending metadata at 1676214034048 (+18 ms)Flushing c54693fd62fafde40475f3c394bfbc2d/cf: closing flushed file at 1676214034048Flushing c54693fd62fafde40475f3c394bfbc2d/cf: reopening flushed file at 1676214034063 (+15 ms)Finished flush of dataSize ~1.11 KB/1137, heapSize ~2.63 KB/2688, currentSize=0 B/0 for c54693fd62fafde40475f3c394bfbc2d in 45ms, sequenceid=8, compaction requested=false at 1676214034066 (+3 ms)Writing region close event to WAL at 1676214034067 (+1 ms)Running coprocessor post-close hooks at 1676214034069 (+2 ms)Closed at 1676214034069 2023-02-12 15:00:34,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:34,071 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=c54693fd62fafde40475f3c394bfbc2d, regionState=CLOSED 2023-02-12 15:00:34,071 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214034071"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214034071"}]},"ts":"1676214034071"} 2023-02-12 15:00:34,072 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:34,072 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=7736b0c91a68e683d42e81f85bf8f471, regionState=CLOSED 2023-02-12 15:00:34,072 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214034072"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214034072"}]},"ts":"1676214034072"} 2023-02-12 15:00:34,073 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=244 2023-02-12 15:00:34,073 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=244, state=SUCCESS; CloseRegionProcedure 1886cfbba21a42a84ae46d52d88f5150, server=jenkins-hbase4.apache.org,38453,1676214004461 in 265 msec 2023-02-12 15:00:34,074 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=246 2023-02-12 15:00:34,074 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=246, state=SUCCESS; CloseRegionProcedure c54693fd62fafde40475f3c394bfbc2d, server=jenkins-hbase4.apache.org,37113,1676214004377 in 265 msec 2023-02-12 15:00:34,075 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, UNASSIGN in 273 msec 2023-02-12 15:00:34,075 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=247 2023-02-12 15:00:34,075 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, UNASSIGN in 274 msec 2023-02-12 15:00:34,075 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=247, state=SUCCESS; CloseRegionProcedure 7736b0c91a68e683d42e81f85bf8f471, server=jenkins-hbase4.apache.org,33845,1676214004423 in 270 msec 2023-02-12 15:00:34,076 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=247, resume processing ppid=242 2023-02-12 15:00:34,077 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, UNASSIGN in 275 msec 2023-02-12 15:00:34,077 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214034077"}]},"ts":"1676214034077"} 2023-02-12 15:00:34,078 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676214031582, state=DISABLED in hbase:meta 2023-02-12 15:00:34,080 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676214031582 to state=DISABLED 2023-02-12 15:00:34,081 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676214031582 in 386 msec 2023-02-12 15:00:34,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-12 15:00:34,449 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676214031582, procId: 242 completed 2023-02-12 15:00:34,450 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676214031582 table=testCorruptedSnapshot-1676214031582 type=FLUSH ttl=0 } 2023-02-12 15:00:34,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676214034450 (current time:1676214034450). 2023-02-12 15:00:34,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-12 15:00:34,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(139): Connect 0x6313dab2 to 127.0.0.1:61893 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-12 15:00:34,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2cfa107f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-12 15:00:34,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-12 15:00:34,459 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40626, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-12 15:00:34,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6313dab2 to 127.0.0.1:61893 2023-02-12 15:00:34,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:34,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-12 15:00:34,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676214031582 table=testCorruptedSnapshot-1676214031582 type=FLUSH ttl=0 } 2023-02-12 15:00:34,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676214031582, type=EXCLUSIVE 2023-02-12 15:00:34,464 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676214031582, type=EXCLUSIVE 2023-02-12 15:00:34,464 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676214031582, type=EXCLUSIVE; timeout=600000, timestamp=1676214634464 2023-02-12 15:00:34,464 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676214031582 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676214031582 2023-02-12 15:00:34,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676214031582 table=testCorruptedSnapshot-1676214031582 type=FLUSH ttl=0 } 2023-02-12 15:00:34,465 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676214031582 table=testCorruptedSnapshot-1676214031582 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-12 15:00:34,465 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-12 15:00:34,479 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676214031582 table=testCorruptedSnapshot-1676214031582 type=DISABLED ttl=0 } 2023-02-12 15:00:34,483 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:34,483 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:34,483 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:34,484 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:34,484 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:34,484 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:34,484 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:34,484 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:34,484 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:34,484 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:34,484 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:34,484 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:34,485 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-12 15:00:34,485 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-12 15:00:34,486 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf/db6bc7be9e7747aabb3aedbe022bfbf2] hfiles 2023-02-12 15:00:34,486 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf/7f393b696f8949d1994c4ab9ad8fa066] hfiles 2023-02-12 15:00:34,486 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf/db6bc7be9e7747aabb3aedbe022bfbf2 2023-02-12 15:00:34,486 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf/7f393b696f8949d1994c4ab9ad8fa066 2023-02-12 15:00:34,486 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/03355b39c2174ea5b1a94a16f11cc6f5] hfiles 2023-02-12 15:00:34,486 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/03355b39c2174ea5b1a94a16f11cc6f5 2023-02-12 15:00:34,487 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf/e63f254e2aa0453ea88616fb006ac4ba] hfiles 2023-02-12 15:00:34,487 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf/e63f254e2aa0453ea88616fb006ac4ba 2023-02-12 15:00:34,488 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf/8548c4a9d6ea4c02a2357194a5e51e1c] hfiles 2023-02-12 15:00:34,488 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf/8548c4a9d6ea4c02a2357194a5e51e1c 2023-02-12 15:00:34,493 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/a87ff679a2f3e71d9181a67b7542122c2023021241aaa89f281348dd86ac054a756c55f2, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c4ca4238a0b923820dcc509a6f75849b20230212610045b633744904b62a524e2fd91898, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c81e728d9d4c2f636f067f89cc14862c202302124cf05442fae6401aa73faf7b5237b337, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/d41d8cd98f00b204e9800998ecf8427e20230212c91709444cd846a4bf112a5d5151b599, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/e4da3b7fbbce2345d7772b0674a318d5202302120e96bf35121a4a4baf068cc3019938dc, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212af309e19f478444b804a53dd73a9b94b] hfiles 2023-02-12 15:00:34,494 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/a87ff679a2f3e71d9181a67b7542122c2023021241aaa89f281348dd86ac054a756c55f2 2023-02-12 15:00:34,494 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c4ca4238a0b923820dcc509a6f75849b20230212610045b633744904b62a524e2fd91898 2023-02-12 15:00:34,494 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c81e728d9d4c2f636f067f89cc14862c202302124cf05442fae6401aa73faf7b5237b337 2023-02-12 15:00:34,494 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/d41d8cd98f00b204e9800998ecf8427e20230212c91709444cd846a4bf112a5d5151b599 2023-02-12 15:00:34,494 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/e4da3b7fbbce2345d7772b0674a318d5202302120e96bf35121a4a4baf068cc3019938dc 2023-02-12 15:00:34,494 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212af309e19f478444b804a53dd73a9b94b 2023-02-12 15:00:34,487 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf/7f593598d537472990eccbd63800672b] hfiles 2023-02-12 15:00:34,496 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf/7f593598d537472990eccbd63800672b 2023-02-12 15:00:34,516 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676214031582 table=testCorruptedSnapshot-1676214031582 type=DISABLED ttl=0 } as finished. 2023-02-12 15:00:34,517 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676214031582 2023-02-12 15:00:34,517 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/snaptb0-1676214031582 2023-02-12 15:00:34,536 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/snaptb0-1676214031582 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/snaptb0-1676214031582 2023-02-12 15:00:34,553 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676214031582 of table testCorruptedSnapshot-1676214031582 completed 2023-02-12 15:00:34,553 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/snaptb0-1676214031582 2023-02-12 15:00:34,553 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.hbase-snapshot/.tmp/snaptb0-1676214031582 2023-02-12 15:00:34,553 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676214031582 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676214031582 at 1676214034464Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676214031582 table=testCorruptedSnapshot-1676214031582 type=DISABLED ttl=0 } at 1676214034480 (+16 ms)Consolidate snapshot: snaptb0-1676214031582 at 1676214034517 (+37 ms)Loading Region manifests for snaptb0-1676214031582 at 1676214034517Writing data manifest for snaptb0-1676214031582 at 1676214034520 (+3 ms)Verifying snapshot: snaptb0-1676214031582 at 1676214034527 (+7 ms)Snapshot snaptb0-1676214031582 of table testCorruptedSnapshot-1676214031582 completed at 1676214034553 (+26 ms) 2023-02-12 15:00:34,556 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676214031582, type=EXCLUSIVE 2023-02-12 15:00:34,557 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676214031582, type=EXCLUSIVE in 94 msec 2023-02-12 15:00:34,715 DEBUG [Listener at localhost/38323] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-12 15:00:34,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676214031582 table=testCorruptedSnapshot-1676214031582 type=FLUSH ttl=0 } is done 2023-02-12 15:00:34,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676214031582 table=testCorruptedSnapshot-1676214031582 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-12 15:00:34,717 INFO [Listener at localhost/38323] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676214031582 2023-02-12 15:00:34,717 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676214031582 2023-02-12 15:00:34,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676214031582 2023-02-12 15:00:34,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-12 15:00:34,720 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676214031582 2023-02-12 15:00:34,720 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214034720"}]},"ts":"1676214034720"} 2023-02-12 15:00:34,721 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676214031582, state=ENABLING in hbase:meta 2023-02-12 15:00:34,723 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-12 15:00:34,724 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, ASSIGN}] 2023-02-12 15:00:34,724 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, ASSIGN 2023-02-12 15:00:34,724 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, ASSIGN 2023-02-12 15:00:34,724 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, ASSIGN 2023-02-12 15:00:34,725 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, ASSIGN 2023-02-12 15:00:34,725 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:34,725 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, ASSIGN 2023-02-12 15:00:34,725 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=true 2023-02-12 15:00:34,725 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:34,725 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=true 2023-02-12 15:00:34,725 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:34,725 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,37113,1676214004377; forceNewPlan=false, retain=true 2023-02-12 15:00:34,726 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:34,726 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=true 2023-02-12 15:00:34,726 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:34,726 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38453,1676214004461; forceNewPlan=false, retain=true 2023-02-12 15:00:34,726 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, ASSIGN 2023-02-12 15:00:34,727 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:34,727 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,33845,1676214004423; forceNewPlan=false, retain=true 2023-02-12 15:00:34,875 INFO [jenkins-hbase4:45369] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-12 15:00:34,876 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=6caf9978ca0972f75c0854f1c9fb76e3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:34,876 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=7736b0c91a68e683d42e81f85bf8f471, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:34,876 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=629e2ec9e8223cc0fb7f0875b15cef8c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:34,876 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214034876"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214034876"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214034876"}]},"ts":"1676214034876"} 2023-02-12 15:00:34,876 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=c54693fd62fafde40475f3c394bfbc2d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:34,876 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=1886cfbba21a42a84ae46d52d88f5150, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:34,876 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214034876"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214034876"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214034876"}]},"ts":"1676214034876"} 2023-02-12 15:00:34,876 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214034876"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214034876"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214034876"}]},"ts":"1676214034876"} 2023-02-12 15:00:34,876 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214034876"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214034876"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214034876"}]},"ts":"1676214034876"} 2023-02-12 15:00:34,876 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214034876"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214034876"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214034876"}]},"ts":"1676214034876"} 2023-02-12 15:00:34,877 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=261, state=RUNNABLE; OpenRegionProcedure 7736b0c91a68e683d42e81f85bf8f471, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:34,878 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=260, state=RUNNABLE; OpenRegionProcedure c54693fd62fafde40475f3c394bfbc2d, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:34,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=258, state=RUNNABLE; OpenRegionProcedure 1886cfbba21a42a84ae46d52d88f5150, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:34,879 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=259, state=RUNNABLE; OpenRegionProcedure 629e2ec9e8223cc0fb7f0875b15cef8c, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:34,880 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=262, state=RUNNABLE; OpenRegionProcedure 6caf9978ca0972f75c0854f1c9fb76e3, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:34,885 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=e5e89645c25979bd82f9e1e324b861b7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:34,885 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214034885"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214034885"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214034885"}]},"ts":"1676214034885"} 2023-02-12 15:00:34,887 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=257, state=RUNNABLE; OpenRegionProcedure e5e89645c25979bd82f9e1e324b861b7, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:34,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-12 15:00:35,037 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:35,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7736b0c91a68e683d42e81f85bf8f471, NAME => 'testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.', STARTKEY => '4', ENDKEY => '5'} 2023-02-12 15:00:35,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:35,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:35,037 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:35,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:35,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:35,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6caf9978ca0972f75c0854f1c9fb76e3, NAME => 'testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.', STARTKEY => '5', ENDKEY => ''} 2023-02-12 15:00:35,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:35,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c54693fd62fafde40475f3c394bfbc2d, NAME => 'testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.', STARTKEY => '3', ENDKEY => '4'} 2023-02-12 15:00:35,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:35,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:35,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:35,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:35,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:35,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:35,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:35,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:35,039 INFO [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:35,039 INFO [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:35,039 INFO [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:35,039 DEBUG [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf 2023-02-12 15:00:35,039 DEBUG [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf 2023-02-12 15:00:35,040 DEBUG [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf 2023-02-12 15:00:35,040 INFO [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7736b0c91a68e683d42e81f85bf8f471 columnFamilyName cf 2023-02-12 15:00:35,040 DEBUG [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf 2023-02-12 15:00:35,040 DEBUG [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf 2023-02-12 15:00:35,040 DEBUG [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf 2023-02-12 15:00:35,040 INFO [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6caf9978ca0972f75c0854f1c9fb76e3 columnFamilyName cf 2023-02-12 15:00:35,040 INFO [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c54693fd62fafde40475f3c394bfbc2d columnFamilyName cf 2023-02-12 15:00:35,043 DEBUG [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf/7f593598d537472990eccbd63800672b 2023-02-12 15:00:35,044 INFO [StoreOpener-7736b0c91a68e683d42e81f85bf8f471-1] regionserver.HStore(310): Store=7736b0c91a68e683d42e81f85bf8f471/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:35,044 DEBUG [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf/e63f254e2aa0453ea88616fb006ac4ba 2023-02-12 15:00:35,044 DEBUG [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf/8548c4a9d6ea4c02a2357194a5e51e1c 2023-02-12 15:00:35,044 INFO [StoreOpener-c54693fd62fafde40475f3c394bfbc2d-1] regionserver.HStore(310): Store=c54693fd62fafde40475f3c394bfbc2d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:35,044 INFO [StoreOpener-6caf9978ca0972f75c0854f1c9fb76e3-1] regionserver.HStore(310): Store=6caf9978ca0972f75c0854f1c9fb76e3/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:35,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:35,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:35,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:35,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:35,045 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:35,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:35,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:35,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:35,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:35,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c54693fd62fafde40475f3c394bfbc2d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69192471, jitterRate=0.031048163771629333}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:35,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c54693fd62fafde40475f3c394bfbc2d: Running coprocessor pre-open hook at 1676214035038Writing region info on filesystem at 1676214035038Initializing all the Stores at 1676214035039 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214035039Cleaning up temporary data from old regions at 1676214035046 (+7 ms)Cleaning up detritus from prior splits at 1676214035046Running coprocessor post-open hooks at 1676214035049 (+3 ms)Region opened successfully at 1676214035049 2023-02-12 15:00:35,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d., pid=264, masterSystemTime=1676214035035 2023-02-12 15:00:35,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:35,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:35,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:35,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 629e2ec9e8223cc0fb7f0875b15cef8c, NAME => 'testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.', STARTKEY => '2', ENDKEY => '3'} 2023-02-12 15:00:35,051 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=c54693fd62fafde40475f3c394bfbc2d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:35,051 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214035051"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214035051"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214035051"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214035051"}]},"ts":"1676214035051"} 2023-02-12 15:00:35,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:35,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:35,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:35,051 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:35,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7736b0c91a68e683d42e81f85bf8f471; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73088869, jitterRate=0.08910901844501495}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:35,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7736b0c91a68e683d42e81f85bf8f471: Running coprocessor pre-open hook at 1676214035038Writing region info on filesystem at 1676214035038Initializing all the Stores at 1676214035038Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214035038Cleaning up temporary data from old regions at 1676214035046 (+8 ms)Cleaning up detritus from prior splits at 1676214035046Running coprocessor post-open hooks at 1676214035052 (+6 ms)Region opened successfully at 1676214035052 2023-02-12 15:00:35,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 6caf9978ca0972f75c0854f1c9fb76e3; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72504762, jitterRate=0.08040514588356018}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:35,052 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 6caf9978ca0972f75c0854f1c9fb76e3: Running coprocessor pre-open hook at 1676214035038Writing region info on filesystem at 1676214035038Initializing all the Stores at 1676214035039 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214035039Cleaning up temporary data from old regions at 1676214035046 (+7 ms)Cleaning up detritus from prior splits at 1676214035046Running coprocessor post-open hooks at 1676214035052 (+6 ms)Region opened successfully at 1676214035052 2023-02-12 15:00:35,053 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471., pid=263, masterSystemTime=1676214035034 2023-02-12 15:00:35,053 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3., pid=267, masterSystemTime=1676214035035 2023-02-12 15:00:35,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:35,054 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:35,054 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:35,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e5e89645c25979bd82f9e1e324b861b7, NAME => 'testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.', STARTKEY => '', ENDKEY => '1'} 2023-02-12 15:00:35,054 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:35,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:35,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:35,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:35,055 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=7736b0c91a68e683d42e81f85bf8f471, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:35,055 INFO [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:35,055 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=260 2023-02-12 15:00:35,055 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214035055"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214035055"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214035055"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214035055"}]},"ts":"1676214035055"} 2023-02-12 15:00:35,056 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:35,056 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:35,055 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=260, state=SUCCESS; OpenRegionProcedure c54693fd62fafde40475f3c394bfbc2d, server=jenkins-hbase4.apache.org,37113,1676214004377 in 175 msec 2023-02-12 15:00:35,056 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:35,056 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1886cfbba21a42a84ae46d52d88f5150, NAME => 'testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.', STARTKEY => '1', ENDKEY => '2'} 2023-02-12 15:00:35,056 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676214031582 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:35,056 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-12 15:00:35,056 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:35,056 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:35,057 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=6caf9978ca0972f75c0854f1c9fb76e3, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:35,057 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214035056"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214035056"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214035056"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214035056"}]},"ts":"1676214035056"} 2023-02-12 15:00:35,057 INFO [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:35,057 INFO [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:35,058 DEBUG [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf 2023-02-12 15:00:35,058 DEBUG [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf 2023-02-12 15:00:35,059 DEBUG [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf 2023-02-12 15:00:35,059 DEBUG [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf 2023-02-12 15:00:35,059 INFO [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1886cfbba21a42a84ae46d52d88f5150 columnFamilyName cf 2023-02-12 15:00:35,059 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, ASSIGN in 331 msec 2023-02-12 15:00:35,059 INFO [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e5e89645c25979bd82f9e1e324b861b7 columnFamilyName cf 2023-02-12 15:00:35,059 DEBUG [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf 2023-02-12 15:00:35,060 DEBUG [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf 2023-02-12 15:00:35,060 INFO [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 629e2ec9e8223cc0fb7f0875b15cef8c columnFamilyName cf 2023-02-12 15:00:35,061 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=261 2023-02-12 15:00:35,061 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=261, state=SUCCESS; OpenRegionProcedure 7736b0c91a68e683d42e81f85bf8f471, server=jenkins-hbase4.apache.org,33845,1676214004423 in 180 msec 2023-02-12 15:00:35,061 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=262 2023-02-12 15:00:35,061 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=262, state=SUCCESS; OpenRegionProcedure 6caf9978ca0972f75c0854f1c9fb76e3, server=jenkins-hbase4.apache.org,38453,1676214004461 in 179 msec 2023-02-12 15:00:35,062 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, ASSIGN in 337 msec 2023-02-12 15:00:35,062 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, ASSIGN in 337 msec 2023-02-12 15:00:35,067 DEBUG [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/03355b39c2174ea5b1a94a16f11cc6f5 2023-02-12 15:00:35,067 DEBUG [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf/db6bc7be9e7747aabb3aedbe022bfbf2 2023-02-12 15:00:35,067 DEBUG [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] regionserver.HStore(539): loaded hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf/7f393b696f8949d1994c4ab9ad8fa066 2023-02-12 15:00:35,067 INFO [StoreOpener-629e2ec9e8223cc0fb7f0875b15cef8c-1] regionserver.HStore(310): Store=629e2ec9e8223cc0fb7f0875b15cef8c/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:35,067 INFO [StoreOpener-e5e89645c25979bd82f9e1e324b861b7-1] regionserver.HStore(310): Store=e5e89645c25979bd82f9e1e324b861b7/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:35,067 INFO [StoreOpener-1886cfbba21a42a84ae46d52d88f5150-1] regionserver.HStore(310): Store=1886cfbba21a42a84ae46d52d88f5150/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-12 15:00:35,068 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:35,068 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:35,068 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:35,069 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:35,069 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:35,069 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:35,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:35,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:35,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:35,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 629e2ec9e8223cc0fb7f0875b15cef8c; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74504069, jitterRate=0.11019714176654816}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:35,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 629e2ec9e8223cc0fb7f0875b15cef8c: Running coprocessor pre-open hook at 1676214035051Writing region info on filesystem at 1676214035051Initializing all the Stores at 1676214035055 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214035055Cleaning up temporary data from old regions at 1676214035069 (+14 ms)Cleaning up detritus from prior splits at 1676214035070 (+1 ms)Running coprocessor post-open hooks at 1676214035072 (+2 ms)Region opened successfully at 1676214035072 2023-02-12 15:00:35,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1886cfbba21a42a84ae46d52d88f5150; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60544983, jitterRate=-0.09780944883823395}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:35,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1886cfbba21a42a84ae46d52d88f5150: Running coprocessor pre-open hook at 1676214035056Writing region info on filesystem at 1676214035056Initializing all the Stores at 1676214035057 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214035057Cleaning up temporary data from old regions at 1676214035069 (+12 ms)Cleaning up detritus from prior splits at 1676214035070 (+1 ms)Running coprocessor post-open hooks at 1676214035072 (+2 ms)Region opened successfully at 1676214035072 2023-02-12 15:00:35,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e5e89645c25979bd82f9e1e324b861b7; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75423525, jitterRate=0.12389810383319855}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-12 15:00:35,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e5e89645c25979bd82f9e1e324b861b7: Running coprocessor pre-open hook at 1676214035055Writing region info on filesystem at 1676214035055Initializing all the Stores at 1676214035055Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676214035055Cleaning up temporary data from old regions at 1676214035069 (+14 ms)Cleaning up detritus from prior splits at 1676214035070 (+1 ms)Running coprocessor post-open hooks at 1676214035072 (+2 ms)Region opened successfully at 1676214035072 2023-02-12 15:00:35,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c., pid=266, masterSystemTime=1676214035035 2023-02-12 15:00:35,073 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150., pid=265, masterSystemTime=1676214035035 2023-02-12 15:00:35,073 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7., pid=268, masterSystemTime=1676214035034 2023-02-12 15:00:35,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:35,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:35,074 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=629e2ec9e8223cc0fb7f0875b15cef8c, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:35,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:35,074 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214035074"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214035074"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214035074"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214035074"}]},"ts":"1676214035074"} 2023-02-12 15:00:35,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:35,075 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=1886cfbba21a42a84ae46d52d88f5150, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:35,075 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214035075"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214035075"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214035075"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214035075"}]},"ts":"1676214035075"} 2023-02-12 15:00:35,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:35,075 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:35,075 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=e5e89645c25979bd82f9e1e324b861b7, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:35,076 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214035075"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676214035075"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676214035075"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676214035075"}]},"ts":"1676214035075"} 2023-02-12 15:00:35,077 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=259 2023-02-12 15:00:35,077 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=259, state=SUCCESS; OpenRegionProcedure 629e2ec9e8223cc0fb7f0875b15cef8c, server=jenkins-hbase4.apache.org,37113,1676214004377 in 197 msec 2023-02-12 15:00:35,078 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=258 2023-02-12 15:00:35,078 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=258, state=SUCCESS; OpenRegionProcedure 1886cfbba21a42a84ae46d52d88f5150, server=jenkins-hbase4.apache.org,38453,1676214004461 in 197 msec 2023-02-12 15:00:35,079 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, ASSIGN in 354 msec 2023-02-12 15:00:35,079 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=257 2023-02-12 15:00:35,079 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=257, state=SUCCESS; OpenRegionProcedure e5e89645c25979bd82f9e1e324b861b7, server=jenkins-hbase4.apache.org,33845,1676214004423 in 190 msec 2023-02-12 15:00:35,079 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, ASSIGN in 354 msec 2023-02-12 15:00:35,080 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=257, resume processing ppid=256 2023-02-12 15:00:35,080 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, ASSIGN in 355 msec 2023-02-12 15:00:35,081 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214035081"}]},"ts":"1676214035081"} 2023-02-12 15:00:35,082 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676214031582, state=ENABLED in hbase:meta 2023-02-12 15:00:35,084 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676214031582' was successfully enabled. 2023-02-12 15:00:35,085 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676214031582 in 367 msec 2023-02-12 15:00:35,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-12 15:00:35,472 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676214031582, procId: 256 completed 2023-02-12 15:00:35,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:35,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:35,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33845] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:35,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37113] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:35,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:35,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38453] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. with WAL disabled. Data may be lost in the event of a crash. 2023-02-12 15:00:35,483 DEBUG [Listener at localhost/38323] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676214031582 2023-02-12 15:00:35,483 DEBUG [Listener at localhost/38323] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:35,483 INFO [Listener at localhost/38323] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-12 15:00:35,603 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-12 15:00:35,701 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-12 15:00:35,786 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-12 15:00:35,824 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-12 15:00:35,912 DEBUG [Listener at localhost/38323] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-12 15:00:36,692 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: 03355b39c2174ea5b1a94a16f11cc6f5 in the real (hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/03355b39c2174ea5b1a94a16f11cc6f5) or archive (hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/03355b39c2174ea5b1a94a16f11cc6f5) directory for the primary table. 2023-02-12 15:00:36,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 03355b39c2174ea5b1a94a16f11cc6f5 in the real (hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/03355b39c2174ea5b1a94a16f11cc6f5) or archive (hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/03355b39c2174ea5b1a94a16f11cc6f5) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2493) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2489) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-12 15:00:36,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] ipc.CallRunner(144): callId: 360 service: MasterService methodName: RestoreSnapshot size: 114 connection: 172.31.14.131:37720 deadline: 1676214096687, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 03355b39c2174ea5b1a94a16f11cc6f5 in the real (hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/03355b39c2174ea5b1a94a16f11cc6f5) or archive (hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/03355b39c2174ea5b1a94a16f11cc6f5) directory for the primary table. 2023-02-12 15:00:36,712 INFO [Listener at localhost/38323] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676214031582 2023-02-12 15:00:36,713 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676214031582 2023-02-12 15:00:36,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676214031582 2023-02-12 15:00:36,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-12 15:00:36,716 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214036716"}]},"ts":"1676214036716"} 2023-02-12 15:00:36,717 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676214031582, state=DISABLING in hbase:meta 2023-02-12 15:00:36,719 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676214031582 to state=DISABLING 2023-02-12 15:00:36,719 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, UNASSIGN}] 2023-02-12 15:00:36,720 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, UNASSIGN 2023-02-12 15:00:36,720 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, UNASSIGN 2023-02-12 15:00:36,720 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, UNASSIGN 2023-02-12 15:00:36,720 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, UNASSIGN 2023-02-12 15:00:36,720 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, UNASSIGN 2023-02-12 15:00:36,721 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=7736b0c91a68e683d42e81f85bf8f471, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:36,721 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=629e2ec9e8223cc0fb7f0875b15cef8c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:36,721 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214036721"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214036721"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214036721"}]},"ts":"1676214036721"} 2023-02-12 15:00:36,721 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=c54693fd62fafde40475f3c394bfbc2d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:36,721 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214036721"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214036721"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214036721"}]},"ts":"1676214036721"} 2023-02-12 15:00:36,721 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214036721"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214036721"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214036721"}]},"ts":"1676214036721"} 2023-02-12 15:00:36,721 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=1886cfbba21a42a84ae46d52d88f5150, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:36,722 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=6caf9978ca0972f75c0854f1c9fb76e3, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:36,722 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214036721"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214036721"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214036721"}]},"ts":"1676214036721"} 2023-02-12 15:00:36,722 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214036722"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214036722"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214036722"}]},"ts":"1676214036722"} 2023-02-12 15:00:36,722 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=275, state=RUNNABLE; CloseRegionProcedure 7736b0c91a68e683d42e81f85bf8f471, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:36,722 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=273, state=RUNNABLE; CloseRegionProcedure 629e2ec9e8223cc0fb7f0875b15cef8c, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:36,723 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=274, state=RUNNABLE; CloseRegionProcedure c54693fd62fafde40475f3c394bfbc2d, server=jenkins-hbase4.apache.org,37113,1676214004377}] 2023-02-12 15:00:36,724 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=272, state=RUNNABLE; CloseRegionProcedure 1886cfbba21a42a84ae46d52d88f5150, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:36,724 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=276, state=RUNNABLE; CloseRegionProcedure 6caf9978ca0972f75c0854f1c9fb76e3, server=jenkins-hbase4.apache.org,38453,1676214004461}] 2023-02-12 15:00:36,724 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, UNASSIGN 2023-02-12 15:00:36,725 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=e5e89645c25979bd82f9e1e324b861b7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:36,725 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214036725"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676214036725"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676214036725"}]},"ts":"1676214036725"} 2023-02-12 15:00:36,726 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=271, state=RUNNABLE; CloseRegionProcedure e5e89645c25979bd82f9e1e324b861b7, server=jenkins-hbase4.apache.org,33845,1676214004423}] 2023-02-12 15:00:36,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:36,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7736b0c91a68e683d42e81f85bf8f471, disabling compactions & flushes 2023-02-12 15:00:36,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:36,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:36,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. after waiting 0 ms 2023-02-12 15:00:36,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:36,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7736b0c91a68e683d42e81f85bf8f471 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-12 15:00:36,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:36,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c54693fd62fafde40475f3c394bfbc2d, disabling compactions & flushes 2023-02-12 15:00:36,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:36,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:36,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. after waiting 0 ms 2023-02-12 15:00:36,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:36,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:36,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c54693fd62fafde40475f3c394bfbc2d 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-12 15:00:36,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 6caf9978ca0972f75c0854f1c9fb76e3, disabling compactions & flushes 2023-02-12 15:00:36,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:36,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:36,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. after waiting 0 ms 2023-02-12 15:00:36,876 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:36,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 6caf9978ca0972f75c0854f1c9fb76e3 1/1 column families, dataSize=22.31 KB heapSize=48.20 KB 2023-02-12 15:00:36,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302129b7ca20e0c174dc3bf1bdcdd439d27e0 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/e4da3b7fbbce2345d7772b0674a318d5202302129b7ca20e0c174dc3bf1bdcdd439d27e0 2023-02-12 15:00:36,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302129f7e1d430a524351bfa7880162cd2443 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302129f7e1d430a524351bfa7880162cd2443 2023-02-12 15:00:36,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302127a4d918cd58b40a58d1ababc32d64f39 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/a87ff679a2f3e71d9181a67b7542122c202302127a4d918cd58b40a58d1ababc32d64f39 2023-02-12 15:00:36,902 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/.tmp/cf/8bfe6bfc4c57489ebde0aa4d06359ed6 2023-02-12 15:00:36,902 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/.tmp/cf/a255caf7c99a465d9587af1020bd9637 2023-02-12 15:00:36,904 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/.tmp/cf/148144257b7b4584b9c1e44f875a4b1e 2023-02-12 15:00:36,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/.tmp/cf/8bfe6bfc4c57489ebde0aa4d06359ed6 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf/8bfe6bfc4c57489ebde0aa4d06359ed6 2023-02-12 15:00:36,906 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/.tmp/cf/a255caf7c99a465d9587af1020bd9637 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf/a255caf7c99a465d9587af1020bd9637 2023-02-12 15:00:36,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/.tmp/cf/148144257b7b4584b9c1e44f875a4b1e as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf/148144257b7b4584b9c1e44f875a4b1e 2023-02-12 15:00:36,908 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf/8bfe6bfc4c57489ebde0aa4d06359ed6, entries=32, sequenceid=15, filesize=10.4 K 2023-02-12 15:00:36,909 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf/a255caf7c99a465d9587af1020bd9637, entries=27, sequenceid=15, filesize=9.6 K 2023-02-12 15:00:36,909 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for c54693fd62fafde40475f3c394bfbc2d in 33ms, sequenceid=15, compaction requested=false 2023-02-12 15:00:36,912 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 7736b0c91a68e683d42e81f85bf8f471 in 38ms, sequenceid=15, compaction requested=false 2023-02-12 15:00:36,912 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf/148144257b7b4584b9c1e44f875a4b1e, entries=341, sequenceid=15, filesize=63.7 K 2023-02-12 15:00:36,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.31 KB/22843, heapSize ~48.19 KB/49344, currentSize=0 B/0 for 6caf9978ca0972f75c0854f1c9fb76e3 in 37ms, sequenceid=15, compaction requested=false 2023-02-12 15:00:36,914 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 15:00:36,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 15:00:36,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. 2023-02-12 15:00:36,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c54693fd62fafde40475f3c394bfbc2d: Waiting for close lock at 1676214036876Running coprocessor pre-close hooks at 1676214036876Disabling compacts and flushes for region at 1676214036876Disabling writes for close at 1676214036876Obtaining lock to block concurrent updates at 1676214036876Preparing flush snapshotting stores in c54693fd62fafde40475f3c394bfbc2d at 1676214036876Finished memstore snapshotting testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676214036876Flushing stores of testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d. at 1676214036877 (+1 ms)Flushing c54693fd62fafde40475f3c394bfbc2d/cf: creating writer at 1676214036878 (+1 ms)Flushing mob file c54693fd62fafde40475f3c394bfbc2d/cf: appending metadata at 1676214036882 (+4 ms)Flushing mob file c54693fd62fafde40475f3c394bfbc2d/cf: closing flushed file at 1676214036882Flushing c54693fd62fafde40475f3c394bfbc2d/cf: appending metadata at 1676214036895 (+13 ms)Flushing c54693fd62fafde40475f3c394bfbc2d/cf: closing flushed file at 1676214036895Flushing c54693fd62fafde40475f3c394bfbc2d/cf: reopening flushed file at 1676214036906 (+11 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for c54693fd62fafde40475f3c394bfbc2d in 33ms, sequenceid=15, compaction requested=false at 1676214036909 (+3 ms)Writing region close event to WAL at 1676214036912 (+3 ms)Running coprocessor post-close hooks at 1676214036915 (+3 ms)Closed at 1676214036915 2023-02-12 15:00:36,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. 2023-02-12 15:00:36,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7736b0c91a68e683d42e81f85bf8f471: Waiting for close lock at 1676214036874Running coprocessor pre-close hooks at 1676214036874Disabling compacts and flushes for region at 1676214036874Disabling writes for close at 1676214036874Obtaining lock to block concurrent updates at 1676214036874Preparing flush snapshotting stores in 7736b0c91a68e683d42e81f85bf8f471 at 1676214036874Finished memstore snapshotting testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676214036874Flushing stores of testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471. at 1676214036875 (+1 ms)Flushing 7736b0c91a68e683d42e81f85bf8f471/cf: creating writer at 1676214036876 (+1 ms)Flushing mob file 7736b0c91a68e683d42e81f85bf8f471/cf: appending metadata at 1676214036881 (+5 ms)Flushing mob file 7736b0c91a68e683d42e81f85bf8f471/cf: closing flushed file at 1676214036881Flushing 7736b0c91a68e683d42e81f85bf8f471/cf: appending metadata at 1676214036895 (+14 ms)Flushing 7736b0c91a68e683d42e81f85bf8f471/cf: closing flushed file at 1676214036895Flushing 7736b0c91a68e683d42e81f85bf8f471/cf: reopening flushed file at 1676214036906 (+11 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 7736b0c91a68e683d42e81f85bf8f471 in 38ms, sequenceid=15, compaction requested=false at 1676214036912 (+6 ms)Writing region close event to WAL at 1676214036912Running coprocessor post-close hooks at 1676214036915 (+3 ms)Closed at 1676214036915 2023-02-12 15:00:36,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 15:00:36,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. 2023-02-12 15:00:36,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 6caf9978ca0972f75c0854f1c9fb76e3: Waiting for close lock at 1676214036876Running coprocessor pre-close hooks at 1676214036876Disabling compacts and flushes for region at 1676214036876Disabling writes for close at 1676214036876Obtaining lock to block concurrent updates at 1676214036876Preparing flush snapshotting stores in 6caf9978ca0972f75c0854f1c9fb76e3 at 1676214036876Finished memstore snapshotting testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3., syncing WAL and waiting on mvcc, flushsize=dataSize=22843, getHeapSize=49344, getOffHeapSize=0, getCellsCount=341 at 1676214036876Flushing stores of testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3. at 1676214036878 (+2 ms)Flushing 6caf9978ca0972f75c0854f1c9fb76e3/cf: creating writer at 1676214036879 (+1 ms)Flushing mob file 6caf9978ca0972f75c0854f1c9fb76e3/cf: appending metadata at 1676214036884 (+5 ms)Flushing mob file 6caf9978ca0972f75c0854f1c9fb76e3/cf: closing flushed file at 1676214036884Flushing 6caf9978ca0972f75c0854f1c9fb76e3/cf: appending metadata at 1676214036895 (+11 ms)Flushing 6caf9978ca0972f75c0854f1c9fb76e3/cf: closing flushed file at 1676214036895Flushing 6caf9978ca0972f75c0854f1c9fb76e3/cf: reopening flushed file at 1676214036907 (+12 ms)Finished flush of dataSize ~22.31 KB/22843, heapSize ~48.19 KB/49344, currentSize=0 B/0 for 6caf9978ca0972f75c0854f1c9fb76e3 in 37ms, sequenceid=15, compaction requested=false at 1676214036913 (+6 ms)Writing region close event to WAL at 1676214036913Running coprocessor post-close hooks at 1676214036916 (+3 ms)Closed at 1676214036916 2023-02-12 15:00:36,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:36,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:36,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 629e2ec9e8223cc0fb7f0875b15cef8c, disabling compactions & flushes 2023-02-12 15:00:36,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:36,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:36,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. after waiting 0 ms 2023-02-12 15:00:36,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:36,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 629e2ec9e8223cc0fb7f0875b15cef8c 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-12 15:00:36,917 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=c54693fd62fafde40475f3c394bfbc2d, regionState=CLOSED 2023-02-12 15:00:36,917 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214036917"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214036917"}]},"ts":"1676214036917"} 2023-02-12 15:00:36,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:36,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:36,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e5e89645c25979bd82f9e1e324b861b7, disabling compactions & flushes 2023-02-12 15:00:36,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:36,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:36,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. after waiting 0 ms 2023-02-12 15:00:36,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:36,918 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e5e89645c25979bd82f9e1e324b861b7 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-12 15:00:36,918 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=7736b0c91a68e683d42e81f85bf8f471, regionState=CLOSED 2023-02-12 15:00:36,919 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214036918"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214036918"}]},"ts":"1676214036918"} 2023-02-12 15:00:36,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:36,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:36,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1886cfbba21a42a84ae46d52d88f5150, disabling compactions & flushes 2023-02-12 15:00:36,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:36,919 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=6caf9978ca0972f75c0854f1c9fb76e3, regionState=CLOSED 2023-02-12 15:00:36,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:36,920 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214036919"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214036919"}]},"ts":"1676214036919"} 2023-02-12 15:00:36,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. after waiting 0 ms 2023-02-12 15:00:36,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:36,920 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1886cfbba21a42a84ae46d52d88f5150 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-12 15:00:36,923 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=274 2023-02-12 15:00:36,923 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=274, state=SUCCESS; CloseRegionProcedure c54693fd62fafde40475f3c394bfbc2d, server=jenkins-hbase4.apache.org,37113,1676214004377 in 197 msec 2023-02-12 15:00:36,932 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=275 2023-02-12 15:00:36,932 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=275, state=SUCCESS; CloseRegionProcedure 7736b0c91a68e683d42e81f85bf8f471, server=jenkins-hbase4.apache.org,33845,1676214004423 in 199 msec 2023-02-12 15:00:36,933 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=276 2023-02-12 15:00:36,933 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=7736b0c91a68e683d42e81f85bf8f471, UNASSIGN in 213 msec 2023-02-12 15:00:36,933 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=276, state=SUCCESS; CloseRegionProcedure 6caf9978ca0972f75c0854f1c9fb76e3, server=jenkins-hbase4.apache.org,38453,1676214004461 in 198 msec 2023-02-12 15:00:36,933 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=c54693fd62fafde40475f3c394bfbc2d, UNASSIGN in 204 msec 2023-02-12 15:00:36,934 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=6caf9978ca0972f75c0854f1c9fb76e3, UNASSIGN in 214 msec 2023-02-12 15:00:36,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c2023021245f09aa8de9e43ee878d7116afaa899a to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c81e728d9d4c2f636f067f89cc14862c2023021245f09aa8de9e43ee878d7116afaa899a 2023-02-12 15:00:36,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023021257075ca5187647de969db21f5009cac4 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c4ca4238a0b923820dcc509a6f75849b2023021257075ca5187647de969db21f5009cac4 2023-02-12 15:00:36,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/.tmp/cf/44c7b3c43a0c467fa7b20322f6f8a2f9 2023-02-12 15:00:36,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/.tmp/cf/44c7b3c43a0c467fa7b20322f6f8a2f9 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/44c7b3c43a0c467fa7b20322f6f8a2f9 2023-02-12 15:00:36,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-12 15:00:36,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/44c7b3c43a0c467fa7b20322f6f8a2f9, entries=36, sequenceid=15, filesize=11.1 K 2023-02-12 15:00:36,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230212949b8457e5ac45ca81e2c25c9a8fde63 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/d41d8cd98f00b204e9800998ecf8427e20230212949b8457e5ac45ca81e2c25c9a8fde63 2023-02-12 15:00:36,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 629e2ec9e8223cc0fb7f0875b15cef8c in 55ms, sequenceid=15, compaction requested=false 2023-02-12 15:00:36,984 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/.tmp/cf/1f7758cfec444329bd252cd434c1b83f 2023-02-12 15:00:36,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 15:00:36,985 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. 2023-02-12 15:00:36,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 629e2ec9e8223cc0fb7f0875b15cef8c: Waiting for close lock at 1676214036917Running coprocessor pre-close hooks at 1676214036917Disabling compacts and flushes for region at 1676214036917Disabling writes for close at 1676214036917Obtaining lock to block concurrent updates at 1676214036917Preparing flush snapshotting stores in 629e2ec9e8223cc0fb7f0875b15cef8c at 1676214036917Finished memstore snapshotting testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676214036917Flushing stores of testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c. at 1676214036918 (+1 ms)Flushing 629e2ec9e8223cc0fb7f0875b15cef8c/cf: creating writer at 1676214036918Flushing mob file 629e2ec9e8223cc0fb7f0875b15cef8c/cf: appending metadata at 1676214036925 (+7 ms)Flushing mob file 629e2ec9e8223cc0fb7f0875b15cef8c/cf: closing flushed file at 1676214036925Flushing 629e2ec9e8223cc0fb7f0875b15cef8c/cf: appending metadata at 1676214036939 (+14 ms)Flushing 629e2ec9e8223cc0fb7f0875b15cef8c/cf: closing flushed file at 1676214036939Flushing 629e2ec9e8223cc0fb7f0875b15cef8c/cf: reopening flushed file at 1676214036965 (+26 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 629e2ec9e8223cc0fb7f0875b15cef8c in 55ms, sequenceid=15, compaction requested=false at 1676214036972 (+7 ms)Writing region close event to WAL at 1676214036977 (+5 ms)Running coprocessor post-close hooks at 1676214036985 (+8 ms)Closed at 1676214036985 2023-02-12 15:00:36,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:36,987 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=629e2ec9e8223cc0fb7f0875b15cef8c, regionState=CLOSED 2023-02-12 15:00:36,987 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214036987"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214036987"}]},"ts":"1676214036987"} 2023-02-12 15:00:36,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/.tmp/cf/1f7758cfec444329bd252cd434c1b83f as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf/1f7758cfec444329bd252cd434c1b83f 2023-02-12 15:00:36,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/.tmp/cf/a49b357ef43243b8b8c86bbfd3bc4027 2023-02-12 15:00:36,993 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=273 2023-02-12 15:00:36,993 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=273, state=SUCCESS; CloseRegionProcedure 629e2ec9e8223cc0fb7f0875b15cef8c, server=jenkins-hbase4.apache.org,37113,1676214004377 in 267 msec 2023-02-12 15:00:36,994 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=629e2ec9e8223cc0fb7f0875b15cef8c, UNASSIGN in 274 msec 2023-02-12 15:00:36,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf/1f7758cfec444329bd252cd434c1b83f, entries=32, sequenceid=15, filesize=10.4 K 2023-02-12 15:00:36,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/.tmp/cf/a49b357ef43243b8b8c86bbfd3bc4027 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf/a49b357ef43243b8b8c86bbfd3bc4027 2023-02-12 15:00:36,996 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 1886cfbba21a42a84ae46d52d88f5150 in 76ms, sequenceid=15, compaction requested=false 2023-02-12 15:00:36,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 15:00:36,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf/a49b357ef43243b8b8c86bbfd3bc4027, entries=32, sequenceid=15, filesize=10.4 K 2023-02-12 15:00:36,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. 2023-02-12 15:00:37,000 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1886cfbba21a42a84ae46d52d88f5150: Waiting for close lock at 1676214036919Running coprocessor pre-close hooks at 1676214036919Disabling compacts and flushes for region at 1676214036919Disabling writes for close at 1676214036920 (+1 ms)Obtaining lock to block concurrent updates at 1676214036920Preparing flush snapshotting stores in 1886cfbba21a42a84ae46d52d88f5150 at 1676214036920Finished memstore snapshotting testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676214036920Flushing stores of testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150. at 1676214036921 (+1 ms)Flushing 1886cfbba21a42a84ae46d52d88f5150/cf: creating writer at 1676214036922 (+1 ms)Flushing mob file 1886cfbba21a42a84ae46d52d88f5150/cf: appending metadata at 1676214036927 (+5 ms)Flushing mob file 1886cfbba21a42a84ae46d52d88f5150/cf: closing flushed file at 1676214036927Flushing 1886cfbba21a42a84ae46d52d88f5150/cf: appending metadata at 1676214036951 (+24 ms)Flushing 1886cfbba21a42a84ae46d52d88f5150/cf: closing flushed file at 1676214036951Flushing 1886cfbba21a42a84ae46d52d88f5150/cf: reopening flushed file at 1676214036992 (+41 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 1886cfbba21a42a84ae46d52d88f5150 in 76ms, sequenceid=15, compaction requested=false at 1676214036996 (+4 ms)Writing region close event to WAL at 1676214036997 (+1 ms)Running coprocessor post-close hooks at 1676214036999 (+2 ms)Closed at 1676214036999 2023-02-12 15:00:37,000 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for e5e89645c25979bd82f9e1e324b861b7 in 83ms, sequenceid=15, compaction requested=false 2023-02-12 15:00:37,001 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:37,001 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=1886cfbba21a42a84ae46d52d88f5150, regionState=CLOSED 2023-02-12 15:00:37,001 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676214037001"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214037001"}]},"ts":"1676214037001"} 2023-02-12 15:00:37,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-12 15:00:37,005 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. 2023-02-12 15:00:37,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e5e89645c25979bd82f9e1e324b861b7: Waiting for close lock at 1676214036917Running coprocessor pre-close hooks at 1676214036917Disabling compacts and flushes for region at 1676214036917Disabling writes for close at 1676214036917Obtaining lock to block concurrent updates at 1676214036918 (+1 ms)Preparing flush snapshotting stores in e5e89645c25979bd82f9e1e324b861b7 at 1676214036918Finished memstore snapshotting testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676214036919 (+1 ms)Flushing stores of testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7. at 1676214036919Flushing e5e89645c25979bd82f9e1e324b861b7/cf: creating writer at 1676214036920 (+1 ms)Flushing mob file e5e89645c25979bd82f9e1e324b861b7/cf: appending metadata at 1676214036942 (+22 ms)Flushing mob file e5e89645c25979bd82f9e1e324b861b7/cf: closing flushed file at 1676214036942Flushing e5e89645c25979bd82f9e1e324b861b7/cf: appending metadata at 1676214036972 (+30 ms)Flushing e5e89645c25979bd82f9e1e324b861b7/cf: closing flushed file at 1676214036972Flushing e5e89645c25979bd82f9e1e324b861b7/cf: reopening flushed file at 1676214036997 (+25 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for e5e89645c25979bd82f9e1e324b861b7 in 83ms, sequenceid=15, compaction requested=false at 1676214037000 (+3 ms)Writing region close event to WAL at 1676214037001 (+1 ms)Running coprocessor post-close hooks at 1676214037005 (+4 ms)Closed at 1676214037005 2023-02-12 15:00:37,005 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=272 2023-02-12 15:00:37,006 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=272, state=SUCCESS; CloseRegionProcedure 1886cfbba21a42a84ae46d52d88f5150, server=jenkins-hbase4.apache.org,38453,1676214004461 in 280 msec 2023-02-12 15:00:37,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:37,007 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=e5e89645c25979bd82f9e1e324b861b7, regionState=CLOSED 2023-02-12 15:00:37,007 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=1886cfbba21a42a84ae46d52d88f5150, UNASSIGN in 287 msec 2023-02-12 15:00:37,007 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676214037007"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676214037007"}]},"ts":"1676214037007"} 2023-02-12 15:00:37,009 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=271 2023-02-12 15:00:37,009 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=271, state=SUCCESS; CloseRegionProcedure e5e89645c25979bd82f9e1e324b861b7, server=jenkins-hbase4.apache.org,33845,1676214004423 in 282 msec 2023-02-12 15:00:37,010 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=271, resume processing ppid=270 2023-02-12 15:00:37,010 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676214031582, region=e5e89645c25979bd82f9e1e324b861b7, UNASSIGN in 290 msec 2023-02-12 15:00:37,011 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676214037011"}]},"ts":"1676214037011"} 2023-02-12 15:00:37,012 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676214031582, state=DISABLED in hbase:meta 2023-02-12 15:00:37,014 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676214031582 to state=DISABLED 2023-02-12 15:00:37,016 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676214031582 in 302 msec 2023-02-12 15:00:37,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-12 15:00:37,468 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676214031582, procId: 270 completed 2023-02-12 15:00:37,469 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testCorruptedSnapshot-1676214031582 2023-02-12 15:00:37,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676214031582 2023-02-12 15:00:37,471 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676214031582 2023-02-12 15:00:37,472 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676214031582 2023-02-12 15:00:37,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-12 15:00:37,478 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:37,478 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:37,478 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:37,478 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:37,478 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:37,478 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:37,480 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/recovered.edits] 2023-02-12 15:00:37,480 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/recovered.edits] 2023-02-12 15:00:37,480 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/recovered.edits] 2023-02-12 15:00:37,481 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/recovered.edits] 2023-02-12 15:00:37,481 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/recovered.edits] 2023-02-12 15:00:37,481 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf, FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/recovered.edits] 2023-02-12 15:00:37,485 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf/7f393b696f8949d1994c4ab9ad8fa066 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf/7f393b696f8949d1994c4ab9ad8fa066 2023-02-12 15:00:37,485 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf/8bfe6bfc4c57489ebde0aa4d06359ed6 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf/8bfe6bfc4c57489ebde0aa4d06359ed6 2023-02-12 15:00:37,486 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf/1f7758cfec444329bd252cd434c1b83f to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf/1f7758cfec444329bd252cd434c1b83f 2023-02-12 15:00:37,486 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf/7f593598d537472990eccbd63800672b to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf/7f593598d537472990eccbd63800672b 2023-02-12 15:00:37,486 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf/148144257b7b4584b9c1e44f875a4b1e to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf/148144257b7b4584b9c1e44f875a4b1e 2023-02-12 15:00:37,486 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/44c7b3c43a0c467fa7b20322f6f8a2f9 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/cf/44c7b3c43a0c467fa7b20322f6f8a2f9 2023-02-12 15:00:37,487 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf/db6bc7be9e7747aabb3aedbe022bfbf2 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/cf/db6bc7be9e7747aabb3aedbe022bfbf2 2023-02-12 15:00:37,488 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf/a255caf7c99a465d9587af1020bd9637 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/cf/a255caf7c99a465d9587af1020bd9637 2023-02-12 15:00:37,488 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf/a49b357ef43243b8b8c86bbfd3bc4027 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/cf/a49b357ef43243b8b8c86bbfd3bc4027 2023-02-12 15:00:37,488 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf/8548c4a9d6ea4c02a2357194a5e51e1c to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/cf/8548c4a9d6ea4c02a2357194a5e51e1c 2023-02-12 15:00:37,488 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf/e63f254e2aa0453ea88616fb006ac4ba to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/cf/e63f254e2aa0453ea88616fb006ac4ba 2023-02-12 15:00:37,490 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/recovered.edits/18.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c/recovered.edits/18.seqid 2023-02-12 15:00:37,491 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/629e2ec9e8223cc0fb7f0875b15cef8c 2023-02-12 15:00:37,491 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/recovered.edits/18.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150/recovered.edits/18.seqid 2023-02-12 15:00:37,491 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/recovered.edits/18.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7/recovered.edits/18.seqid 2023-02-12 15:00:37,492 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/recovered.edits/18.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3/recovered.edits/18.seqid 2023-02-12 15:00:37,492 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/recovered.edits/18.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471/recovered.edits/18.seqid 2023-02-12 15:00:37,492 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/e5e89645c25979bd82f9e1e324b861b7 2023-02-12 15:00:37,492 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/recovered.edits/18.seqid to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d/recovered.edits/18.seqid 2023-02-12 15:00:37,493 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/1886cfbba21a42a84ae46d52d88f5150 2023-02-12 15:00:37,493 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/6caf9978ca0972f75c0854f1c9fb76e3 2023-02-12 15:00:37,493 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/7736b0c91a68e683d42e81f85bf8f471 2023-02-12 15:00:37,493 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/.tmp/data/default/testCorruptedSnapshot-1676214031582/c54693fd62fafde40475f3c394bfbc2d 2023-02-12 15:00:37,493 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676214031582 regions 2023-02-12 15:00:37,493 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d 2023-02-12 15:00:37,494 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf] 2023-02-12 15:00:37,497 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/a87ff679a2f3e71d9181a67b7542122c2023021241aaa89f281348dd86ac054a756c55f2 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/a87ff679a2f3e71d9181a67b7542122c2023021241aaa89f281348dd86ac054a756c55f2 2023-02-12 15:00:37,498 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/a87ff679a2f3e71d9181a67b7542122c202302127a4d918cd58b40a58d1ababc32d64f39 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/a87ff679a2f3e71d9181a67b7542122c202302127a4d918cd58b40a58d1ababc32d64f39 2023-02-12 15:00:37,499 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c4ca4238a0b923820dcc509a6f75849b2023021257075ca5187647de969db21f5009cac4 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c4ca4238a0b923820dcc509a6f75849b2023021257075ca5187647de969db21f5009cac4 2023-02-12 15:00:37,499 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c4ca4238a0b923820dcc509a6f75849b20230212610045b633744904b62a524e2fd91898 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c4ca4238a0b923820dcc509a6f75849b20230212610045b633744904b62a524e2fd91898 2023-02-12 15:00:37,501 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c81e728d9d4c2f636f067f89cc14862c2023021245f09aa8de9e43ee878d7116afaa899a to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c81e728d9d4c2f636f067f89cc14862c2023021245f09aa8de9e43ee878d7116afaa899a 2023-02-12 15:00:37,501 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c81e728d9d4c2f636f067f89cc14862c202302124cf05442fae6401aa73faf7b5237b337 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/c81e728d9d4c2f636f067f89cc14862c202302124cf05442fae6401aa73faf7b5237b337 2023-02-12 15:00:37,502 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/d41d8cd98f00b204e9800998ecf8427e20230212949b8457e5ac45ca81e2c25c9a8fde63 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/d41d8cd98f00b204e9800998ecf8427e20230212949b8457e5ac45ca81e2c25c9a8fde63 2023-02-12 15:00:37,503 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/d41d8cd98f00b204e9800998ecf8427e20230212c91709444cd846a4bf112a5d5151b599 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/d41d8cd98f00b204e9800998ecf8427e20230212c91709444cd846a4bf112a5d5151b599 2023-02-12 15:00:37,504 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/e4da3b7fbbce2345d7772b0674a318d5202302120e96bf35121a4a4baf068cc3019938dc to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/e4da3b7fbbce2345d7772b0674a318d5202302120e96bf35121a4a4baf068cc3019938dc 2023-02-12 15:00:37,505 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-12 15:00:37,505 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/e4da3b7fbbce2345d7772b0674a318d5202302129b7ca20e0c174dc3bf1bdcdd439d27e0 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/e4da3b7fbbce2345d7772b0674a318d5202302129b7ca20e0c174dc3bf1bdcdd439d27e0 2023-02-12 15:00:37,506 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302129f7e1d430a524351bfa7880162cd2443 to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302129f7e1d430a524351bfa7880162cd2443 2023-02-12 15:00:37,507 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212af309e19f478444b804a53dd73a9b94b to hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/archive/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230212af309e19f478444b804a53dd73a9b94b 2023-02-12 15:00:37,507 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/mobdir/data/default/testCorruptedSnapshot-1676214031582/31999c63042180af1a7898963ccf1e1d 2023-02-12 15:00:37,509 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676214031582 2023-02-12 15:00:37,511 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676214031582 from hbase:meta 2023-02-12 15:00:37,514 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676214031582' descriptor. 2023-02-12 15:00:37,514 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676214031582 2023-02-12 15:00:37,514 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676214031582' from region states. 2023-02-12 15:00:37,515 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214037515"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:37,515 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214037515"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:37,515 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214037515"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:37,515 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214037515"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:37,515 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214037515"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:37,515 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676214037515"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:37,516 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-12 15:00:37,516 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e5e89645c25979bd82f9e1e324b861b7, NAME => 'testCorruptedSnapshot-1676214031582,,1676214031584.e5e89645c25979bd82f9e1e324b861b7.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 1886cfbba21a42a84ae46d52d88f5150, NAME => 'testCorruptedSnapshot-1676214031582,1,1676214031584.1886cfbba21a42a84ae46d52d88f5150.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 629e2ec9e8223cc0fb7f0875b15cef8c, NAME => 'testCorruptedSnapshot-1676214031582,2,1676214031584.629e2ec9e8223cc0fb7f0875b15cef8c.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => c54693fd62fafde40475f3c394bfbc2d, NAME => 'testCorruptedSnapshot-1676214031582,3,1676214031584.c54693fd62fafde40475f3c394bfbc2d.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 7736b0c91a68e683d42e81f85bf8f471, NAME => 'testCorruptedSnapshot-1676214031582,4,1676214031584.7736b0c91a68e683d42e81f85bf8f471.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 6caf9978ca0972f75c0854f1c9fb76e3, NAME => 'testCorruptedSnapshot-1676214031582,5,1676214031584.6caf9978ca0972f75c0854f1c9fb76e3.', STARTKEY => '5', ENDKEY => ''}] 2023-02-12 15:00:37,516 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676214031582' as deleted. 2023-02-12 15:00:37,516 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676214031582","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676214037516"}]},"ts":"9223372036854775807"} 2023-02-12 15:00:37,518 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676214031582 state from META 2023-02-12 15:00:37,519 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676214031582 2023-02-12 15:00:37,520 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676214031582 in 50 msec 2023-02-12 15:00:37,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-12 15:00:37,723 INFO [Listener at localhost/38323] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676214031582, procId: 283 completed 2023-02-12 15:00:37,727 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676214031582" 2023-02-12 15:00:37,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676214031582 2023-02-12 15:00:37,729 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676214031582" 2023-02-12 15:00:37,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45369] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676214031582 2023-02-12 15:00:37,748 INFO [Listener at localhost/38323] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=421 (was 426), OpenFileDescriptor=801 (was 826), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=172 (was 186), ProcessCount=168 (was 168), AvailableMemoryMB=3541 (was 3593) 2023-02-12 15:00:37,748 INFO [Listener at localhost/38323] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-12 15:00:37,748 INFO [Listener at localhost/38323] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-12 15:00:37,749 DEBUG [Listener at localhost/38323] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c633018 to 127.0.0.1:61893 2023-02-12 15:00:37,749 DEBUG [Listener at localhost/38323] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:37,749 DEBUG [Listener at localhost/38323] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-12 15:00:37,749 DEBUG [Listener at localhost/38323] util.JVMClusterUtil(257): Found active master hash=1143794569, stopped=false 2023-02-12 15:00:37,749 INFO [Listener at localhost/38323] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,45369,1676214003194 2023-02-12 15:00:37,752 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-12 15:00:37,752 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-12 15:00:37,752 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-12 15:00:37,752 INFO [Listener at localhost/38323] procedure2.ProcedureExecutor(629): Stopping 2023-02-12 15:00:37,752 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-12 15:00:37,752 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 15:00:37,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 15:00:37,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 15:00:37,753 DEBUG [Listener at localhost/38323] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21deb758 to 127.0.0.1:61893 2023-02-12 15:00:37,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 15:00:37,753 DEBUG [Listener at localhost/38323] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:37,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-12 15:00:37,753 INFO [Listener at localhost/38323] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37113,1676214004377' ***** 2023-02-12 15:00:37,753 INFO [Listener at localhost/38323] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-12 15:00:37,753 INFO [Listener at localhost/38323] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33845,1676214004423' ***** 2023-02-12 15:00:37,753 INFO [Listener at localhost/38323] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-12 15:00:37,753 INFO [Listener at localhost/38323] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38453,1676214004461' ***** 2023-02-12 15:00:37,753 INFO [RS:0;jenkins-hbase4:37113] regionserver.HeapMemoryManager(220): Stopping 2023-02-12 15:00:37,753 INFO [Listener at localhost/38323] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-12 15:00:37,753 INFO [RS:0;jenkins-hbase4:37113] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-12 15:00:37,754 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-12 15:00:37,754 INFO [RS:0;jenkins-hbase4:37113] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-12 15:00:37,754 INFO [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:37,754 DEBUG [RS:0;jenkins-hbase4:37113] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5361090a to 127.0.0.1:61893 2023-02-12 15:00:37,754 INFO [RS:1;jenkins-hbase4:33845] regionserver.HeapMemoryManager(220): Stopping 2023-02-12 15:00:37,754 DEBUG [RS:0;jenkins-hbase4:37113] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:37,754 INFO [RS:1;jenkins-hbase4:33845] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-12 15:00:37,754 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-12 15:00:37,754 INFO [RS:0;jenkins-hbase4:37113] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-12 15:00:37,754 INFO [RS:1;jenkins-hbase4:33845] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-12 15:00:37,754 INFO [RS:2;jenkins-hbase4:38453] regionserver.HeapMemoryManager(220): Stopping 2023-02-12 15:00:37,754 INFO [RS:0;jenkins-hbase4:37113] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-12 15:00:37,754 INFO [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(3304): Received CLOSE for 2e9d615367e89c5413b0d2c088c828a2 2023-02-12 15:00:37,754 INFO [RS:0;jenkins-hbase4:37113] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-12 15:00:37,754 INFO [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-12 15:00:37,754 INFO [RS:2;jenkins-hbase4:38453] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-12 15:00:37,755 INFO [RS:2;jenkins-hbase4:38453] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-12 15:00:37,755 INFO [RS:2;jenkins-hbase4:38453] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:37,755 DEBUG [RS:2;jenkins-hbase4:38453] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ef5b80c to 127.0.0.1:61893 2023-02-12 15:00:37,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 2e9d615367e89c5413b0d2c088c828a2, disabling compactions & flushes 2023-02-12 15:00:37,755 INFO [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-12 15:00:37,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. 2023-02-12 15:00:37,755 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-12 15:00:37,754 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-12 15:00:37,755 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-12 15:00:37,755 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-12 15:00:37,755 DEBUG [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-12 15:00:37,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. 2023-02-12 15:00:37,755 DEBUG [RS:2;jenkins-hbase4:38453] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:37,755 INFO [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:37,755 INFO [RS:2;jenkins-hbase4:38453] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38453,1676214004461; all regions closed. 2023-02-12 15:00:37,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. after waiting 0 ms 2023-02-12 15:00:37,755 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-12 15:00:37,755 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. 2023-02-12 15:00:37,755 DEBUG [RS:1;jenkins-hbase4:33845] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3abc1df0 to 127.0.0.1:61893 2023-02-12 15:00:37,755 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 2e9d615367e89c5413b0d2c088c828a2 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-12 15:00:37,755 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-12 15:00:37,755 DEBUG [RS:1;jenkins-hbase4:33845] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:37,756 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.71 KB heapSize=212.48 KB 2023-02-12 15:00:37,756 INFO [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-12 15:00:37,756 DEBUG [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(1479): Online Regions={2e9d615367e89c5413b0d2c088c828a2=hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2.} 2023-02-12 15:00:37,757 DEBUG [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-12 15:00:37,757 DEBUG [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(1505): Waiting on 2e9d615367e89c5413b0d2c088c828a2 2023-02-12 15:00:37,766 DEBUG [RS:2;jenkins-hbase4:38453] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/oldWALs 2023-02-12 15:00:37,766 INFO [RS:2;jenkins-hbase4:38453] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38453%2C1676214004461:(num 1676214005970) 2023-02-12 15:00:37,767 DEBUG [RS:2;jenkins-hbase4:38453] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:37,767 INFO [RS:2;jenkins-hbase4:38453] regionserver.LeaseManager(133): Closed leases 2023-02-12 15:00:37,772 INFO [RS:2;jenkins-hbase4:38453] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-12 15:00:37,772 INFO [RS:2;jenkins-hbase4:38453] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-12 15:00:37,772 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-12 15:00:37,772 INFO [RS:2;jenkins-hbase4:38453] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-12 15:00:37,773 INFO [RS:2;jenkins-hbase4:38453] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-12 15:00:37,776 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/namespace/2e9d615367e89c5413b0d2c088c828a2/.tmp/info/10350e14990f4736878a6067afe78ea3 2023-02-12 15:00:37,779 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/namespace/2e9d615367e89c5413b0d2c088c828a2/.tmp/info/10350e14990f4736878a6067afe78ea3 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/namespace/2e9d615367e89c5413b0d2c088c828a2/info/10350e14990f4736878a6067afe78ea3 2023-02-12 15:00:37,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/namespace/2e9d615367e89c5413b0d2c088c828a2/info/10350e14990f4736878a6067afe78ea3, entries=2, sequenceid=6, filesize=4.8 K 2023-02-12 15:00:37,783 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 2e9d615367e89c5413b0d2c088c828a2 in 28ms, sequenceid=6, compaction requested=false 2023-02-12 15:00:37,789 INFO [RS:2;jenkins-hbase4:38453] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38453 2023-02-12 15:00:37,791 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=130.94 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/.tmp/info/3bf60443107a478e9ecdb3e318da9fc2 2023-02-12 15:00:37,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/namespace/2e9d615367e89c5413b0d2c088c828a2/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-12 15:00:37,793 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. 2023-02-12 15:00:37,794 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 2e9d615367e89c5413b0d2c088c828a2: Waiting for close lock at 1676214037755Running coprocessor pre-close hooks at 1676214037755Disabling compacts and flushes for region at 1676214037755Disabling writes for close at 1676214037755Obtaining lock to block concurrent updates at 1676214037756 (+1 ms)Preparing flush snapshotting stores in 2e9d615367e89c5413b0d2c088c828a2 at 1676214037756Finished memstore snapshotting hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676214037756Flushing stores of hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. at 1676214037757 (+1 ms)Flushing 2e9d615367e89c5413b0d2c088c828a2/info: creating writer at 1676214037757Flushing 2e9d615367e89c5413b0d2c088c828a2/info: appending metadata at 1676214037762 (+5 ms)Flushing 2e9d615367e89c5413b0d2c088c828a2/info: closing flushed file at 1676214037762Flushing 2e9d615367e89c5413b0d2c088c828a2/info: reopening flushed file at 1676214037780 (+18 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 2e9d615367e89c5413b0d2c088c828a2 in 28ms, sequenceid=6, compaction requested=false at 1676214037783 (+3 ms)Writing region close event to WAL at 1676214037789 (+6 ms)Running coprocessor post-close hooks at 1676214037793 (+4 ms)Closed at 1676214037793 2023-02-12 15:00:37,794 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676214006438.2e9d615367e89c5413b0d2c088c828a2. 2023-02-12 15:00:37,799 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:37,799 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-12 15:00:37,799 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:37,799 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3bf60443107a478e9ecdb3e318da9fc2 2023-02-12 15:00:37,799 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-12 15:00:37,799 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38453,1676214004461 2023-02-12 15:00:37,799 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-12 15:00:37,799 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-12 15:00:37,800 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,38453,1676214004461] 2023-02-12 15:00:37,800 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,38453,1676214004461; numProcessing=1 2023-02-12 15:00:37,803 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,38453,1676214004461 already deleted, retry=false 2023-02-12 15:00:37,803 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,38453,1676214004461 expired; onlineServers=2 2023-02-12 15:00:37,803 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-12 15:00:37,803 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-12 15:00:37,810 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/.tmp/rep_barrier/addedc0d7a9c4202a6243b1b32f49bac 2023-02-12 15:00:37,812 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-12 15:00:37,813 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-12 15:00:37,813 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for addedc0d7a9c4202a6243b1b32f49bac 2023-02-12 15:00:37,821 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/.tmp/table/64b713b4fc5341c090c1c20efbd49249 2023-02-12 15:00:37,824 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 64b713b4fc5341c090c1c20efbd49249 2023-02-12 15:00:37,824 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/.tmp/info/3bf60443107a478e9ecdb3e318da9fc2 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/info/3bf60443107a478e9ecdb3e318da9fc2 2023-02-12 15:00:37,827 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3bf60443107a478e9ecdb3e318da9fc2 2023-02-12 15:00:37,827 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/info/3bf60443107a478e9ecdb3e318da9fc2, entries=64, sequenceid=307, filesize=11.8 K 2023-02-12 15:00:37,828 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/.tmp/rep_barrier/addedc0d7a9c4202a6243b1b32f49bac as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/rep_barrier/addedc0d7a9c4202a6243b1b32f49bac 2023-02-12 15:00:37,830 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for addedc0d7a9c4202a6243b1b32f49bac 2023-02-12 15:00:37,830 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/rep_barrier/addedc0d7a9c4202a6243b1b32f49bac, entries=18, sequenceid=307, filesize=7.0 K 2023-02-12 15:00:37,831 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/.tmp/table/64b713b4fc5341c090c1c20efbd49249 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/table/64b713b4fc5341c090c1c20efbd49249 2023-02-12 15:00:37,833 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 64b713b4fc5341c090c1c20efbd49249 2023-02-12 15:00:37,833 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/table/64b713b4fc5341c090c1c20efbd49249, entries=22, sequenceid=307, filesize=7.0 K 2023-02-12 15:00:37,834 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 78ms, sequenceid=307, compaction requested=false 2023-02-12 15:00:37,837 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-12 15:00:37,838 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-12 15:00:37,839 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-12 15:00:37,839 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676214037755Running coprocessor pre-close hooks at 1676214037755Disabling compacts and flushes for region at 1676214037755Disabling writes for close at 1676214037755Obtaining lock to block concurrent updates at 1676214037756 (+1 ms)Preparing flush snapshotting stores in 1588230740 at 1676214037756Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=141010, getHeapSize=217528, getOffHeapSize=0, getCellsCount=997 at 1676214037756Flushing stores of hbase:meta,,1.1588230740 at 1676214037757 (+1 ms)Flushing 1588230740/info: creating writer at 1676214037757Flushing 1588230740/info: appending metadata at 1676214037764 (+7 ms)Flushing 1588230740/info: closing flushed file at 1676214037764Flushing 1588230740/rep_barrier: creating writer at 1676214037799 (+35 ms)Flushing 1588230740/rep_barrier: appending metadata at 1676214037801 (+2 ms)Flushing 1588230740/rep_barrier: closing flushed file at 1676214037801Flushing 1588230740/table: creating writer at 1676214037813 (+12 ms)Flushing 1588230740/table: appending metadata at 1676214037815 (+2 ms)Flushing 1588230740/table: closing flushed file at 1676214037815Flushing 1588230740/info: reopening flushed file at 1676214037825 (+10 ms)Flushing 1588230740/rep_barrier: reopening flushed file at 1676214037828 (+3 ms)Flushing 1588230740/table: reopening flushed file at 1676214037831 (+3 ms)Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 78ms, sequenceid=307, compaction requested=false at 1676214037834 (+3 ms)Writing region close event to WAL at 1676214037835 (+1 ms)Running coprocessor post-close hooks at 1676214037838 (+3 ms)Closed at 1676214037839 (+1 ms) 2023-02-12 15:00:37,839 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-12 15:00:37,849 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-12 15:00:37,850 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-12 15:00:37,850 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-12 15:00:37,951 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 15:00:37,951 INFO [RS:2;jenkins-hbase4:38453] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38453,1676214004461; zookeeper connection closed. 2023-02-12 15:00:37,951 ERROR [Listener at localhost/38323-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7b8d2e50 rejected from java.util.concurrent.ThreadPoolExecutor@7fb2b5f9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 15:00:37,952 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@52b2e739] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@52b2e739 2023-02-12 15:00:37,952 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:38453-0x100490bcc700003, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 15:00:37,952 ERROR [Listener at localhost/38323-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@192d1b22 rejected from java.util.concurrent.ThreadPoolExecutor@7fb2b5f9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 15:00:37,957 INFO [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37113,1676214004377; all regions closed. 2023-02-12 15:00:37,957 INFO [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33845,1676214004423; all regions closed. 2023-02-12 15:00:37,963 DEBUG [RS:1;jenkins-hbase4:33845] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/oldWALs 2023-02-12 15:00:37,963 DEBUG [RS:0;jenkins-hbase4:37113] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/oldWALs 2023-02-12 15:00:37,963 INFO [RS:1;jenkins-hbase4:33845] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33845%2C1676214004423:(num 1676214006000) 2023-02-12 15:00:37,964 DEBUG [RS:1;jenkins-hbase4:33845] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:37,964 INFO [RS:0;jenkins-hbase4:37113] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37113%2C1676214004377.meta:.meta(num 1676214006162) 2023-02-12 15:00:37,964 INFO [RS:1;jenkins-hbase4:33845] regionserver.LeaseManager(133): Closed leases 2023-02-12 15:00:37,964 INFO [RS:1;jenkins-hbase4:33845] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-12 15:00:37,965 INFO [RS:1;jenkins-hbase4:33845] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-12 15:00:37,965 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-12 15:00:37,965 INFO [RS:1;jenkins-hbase4:33845] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-12 15:00:37,965 INFO [RS:1;jenkins-hbase4:33845] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-12 15:00:37,966 INFO [RS:1;jenkins-hbase4:33845] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33845 2023-02-12 15:00:37,968 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/WALs/jenkins-hbase4.apache.org,37113,1676214004377/jenkins-hbase4.apache.org%2C37113%2C1676214004377.1676214005970 not finished, retry = 0 2023-02-12 15:00:37,970 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-12 15:00:37,970 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:37,970 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33845,1676214004423 2023-02-12 15:00:37,970 ERROR [Listener at localhost/38323-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2690b4f4 rejected from java.util.concurrent.ThreadPoolExecutor@22af24fe[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 15:00:37,971 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,33845,1676214004423] 2023-02-12 15:00:37,972 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33845,1676214004423; numProcessing=2 2023-02-12 15:00:37,973 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33845,1676214004423 already deleted, retry=false 2023-02-12 15:00:37,973 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,33845,1676214004423 expired; onlineServers=1 2023-02-12 15:00:38,071 DEBUG [RS:0;jenkins-hbase4:37113] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/oldWALs 2023-02-12 15:00:38,071 INFO [RS:0;jenkins-hbase4:37113] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37113%2C1676214004377:(num 1676214005970) 2023-02-12 15:00:38,071 DEBUG [RS:0;jenkins-hbase4:37113] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:38,071 INFO [RS:0;jenkins-hbase4:37113] regionserver.LeaseManager(133): Closed leases 2023-02-12 15:00:38,071 INFO [RS:0;jenkins-hbase4:37113] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-12 15:00:38,071 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-12 15:00:38,072 INFO [RS:0;jenkins-hbase4:37113] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37113 2023-02-12 15:00:38,076 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37113,1676214004377 2023-02-12 15:00:38,076 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-12 15:00:38,076 ERROR [Listener at localhost/38323-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@c179bc3 rejected from java.util.concurrent.ThreadPoolExecutor@8281e99[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 15:00:38,077 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,37113,1676214004377] 2023-02-12 15:00:38,077 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,37113,1676214004377; numProcessing=3 2023-02-12 15:00:38,078 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,37113,1676214004377 already deleted, retry=false 2023-02-12 15:00:38,078 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,37113,1676214004377 expired; onlineServers=0 2023-02-12 15:00:38,078 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,45369,1676214003194' ***** 2023-02-12 15:00:38,078 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-12 15:00:38,078 DEBUG [M:0;jenkins-hbase4:45369] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@511e1b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-12 15:00:38,078 INFO [M:0;jenkins-hbase4:45369] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,45369,1676214003194 2023-02-12 15:00:38,078 INFO [M:0;jenkins-hbase4:45369] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,45369,1676214003194; all regions closed. 2023-02-12 15:00:38,079 DEBUG [M:0;jenkins-hbase4:45369] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-12 15:00:38,079 DEBUG [M:0;jenkins-hbase4:45369] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-12 15:00:38,079 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-12 15:00:38,079 DEBUG [M:0;jenkins-hbase4:45369] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-12 15:00:38,079 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676214005620] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676214005620,5,FailOnTimeoutGroup] 2023-02-12 15:00:38,079 INFO [M:0;jenkins-hbase4:45369] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-12 15:00:38,079 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676214005616] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676214005616,5,FailOnTimeoutGroup] 2023-02-12 15:00:38,079 INFO [M:0;jenkins-hbase4:45369] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-12 15:00:38,080 INFO [M:0;jenkins-hbase4:45369] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-12 15:00:38,080 DEBUG [M:0;jenkins-hbase4:45369] master.HMaster(1502): Stopping service threads 2023-02-12 15:00:38,080 INFO [M:0;jenkins-hbase4:45369] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-12 15:00:38,080 ERROR [M:0;jenkins-hbase4:45369] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-12 15:00:38,080 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-12 15:00:38,080 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-12 15:00:38,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-12 15:00:38,081 INFO [M:0;jenkins-hbase4:45369] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-12 15:00:38,081 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-12 15:00:38,081 DEBUG [M:0;jenkins-hbase4:45369] zookeeper.ZKUtil(398): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-12 15:00:38,082 WARN [M:0;jenkins-hbase4:45369] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-12 15:00:38,082 INFO [M:0;jenkins-hbase4:45369] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-12 15:00:38,082 INFO [M:0;jenkins-hbase4:45369] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-12 15:00:38,082 DEBUG [M:0;jenkins-hbase4:45369] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-12 15:00:38,082 INFO [M:0;jenkins-hbase4:45369] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 15:00:38,082 DEBUG [M:0;jenkins-hbase4:45369] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 15:00:38,082 DEBUG [M:0;jenkins-hbase4:45369] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-12 15:00:38,082 DEBUG [M:0;jenkins-hbase4:45369] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 15:00:38,082 INFO [M:0;jenkins-hbase4:45369] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=845.52 KB heapSize=1.01 MB 2023-02-12 15:00:38,095 INFO [M:0;jenkins-hbase4:45369] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=845.52 KB at sequenceid=2120 (bloomFilter=true), to=hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c492ebed4c864e66a1553f09976e42b5 2023-02-12 15:00:38,097 INFO [M:0;jenkins-hbase4:45369] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c492ebed4c864e66a1553f09976e42b5 2023-02-12 15:00:38,098 DEBUG [M:0;jenkins-hbase4:45369] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c492ebed4c864e66a1553f09976e42b5 as hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c492ebed4c864e66a1553f09976e42b5 2023-02-12 15:00:38,100 INFO [M:0;jenkins-hbase4:45369] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c492ebed4c864e66a1553f09976e42b5 2023-02-12 15:00:38,100 INFO [M:0;jenkins-hbase4:45369] regionserver.HStore(1080): Added hdfs://localhost:35339/user/jenkins/test-data/05aeecf7-8a84-625f-dc93-a424e2d8a46b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c492ebed4c864e66a1553f09976e42b5, entries=282, sequenceid=2120, filesize=30.8 K 2023-02-12 15:00:38,101 INFO [M:0;jenkins-hbase4:45369] regionserver.HRegion(2947): Finished flush of dataSize ~845.52 KB/865810, heapSize ~1.01 MB/1062672, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 19ms, sequenceid=2120, compaction requested=false 2023-02-12 15:00:38,102 INFO [M:0;jenkins-hbase4:45369] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-12 15:00:38,102 DEBUG [M:0;jenkins-hbase4:45369] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676214038082Disabling compacts and flushes for region at 1676214038082Disabling writes for close at 1676214038082Obtaining lock to block concurrent updates at 1676214038082Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676214038082Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=865810, getHeapSize=1062672, getOffHeapSize=0, getCellsCount=2607 at 1676214038082Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676214038083 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676214038083Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676214038087 (+4 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676214038087Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676214038098 (+11 ms)Finished flush of dataSize ~845.52 KB/865810, heapSize ~1.01 MB/1062672, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 19ms, sequenceid=2120, compaction requested=false at 1676214038101 (+3 ms)Writing region close event to WAL at 1676214038102 (+1 ms)Closed at 1676214038102 2023-02-12 15:00:38,105 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-12 15:00:38,105 INFO [M:0;jenkins-hbase4:45369] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-12 15:00:38,106 INFO [M:0;jenkins-hbase4:45369] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45369 2023-02-12 15:00:38,109 DEBUG [M:0;jenkins-hbase4:45369] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,45369,1676214003194 already deleted, retry=false 2023-02-12 15:00:38,452 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 15:00:38,452 INFO [M:0;jenkins-hbase4:45369] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,45369,1676214003194; zookeeper connection closed. 2023-02-12 15:00:38,452 ERROR [Listener at localhost/38323-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@31d8ca56 rejected from java.util.concurrent.ThreadPoolExecutor@71c1cc88[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 15:00:38,452 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): master:45369-0x100490bcc700000, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 15:00:38,453 ERROR [Listener at localhost/38323-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@62e2d6f3 rejected from java.util.concurrent.ThreadPoolExecutor@71c1cc88[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 15:00:38,552 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 15:00:38,552 INFO [RS:0;jenkins-hbase4:37113] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37113,1676214004377; zookeeper connection closed. 2023-02-12 15:00:38,552 ERROR [Listener at localhost/38323-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4c54da89 rejected from java.util.concurrent.ThreadPoolExecutor@8281e99[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 15:00:38,553 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:37113-0x100490bcc700001, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 15:00:38,553 ERROR [Listener at localhost/38323-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@34b98c89 rejected from java.util.concurrent.ThreadPoolExecutor@8281e99[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 15:00:38,553 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@47ecd464] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@47ecd464 2023-02-12 15:00:38,653 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 15:00:38,653 INFO [RS:1;jenkins-hbase4:33845] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33845,1676214004423; zookeeper connection closed. 2023-02-12 15:00:38,653 ERROR [Listener at localhost/38323-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7514b68b rejected from java.util.concurrent.ThreadPoolExecutor@22af24fe[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 15:00:38,653 DEBUG [Listener at localhost/38323-EventThread] zookeeper.ZKWatcher(600): regionserver:33845-0x100490bcc700002, quorum=127.0.0.1:61893, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-12 15:00:38,653 ERROR [Listener at localhost/38323-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@d53cd23 rejected from java.util.concurrent.ThreadPoolExecutor@22af24fe[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-12 15:00:38,653 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6cc1b968] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6cc1b968 2023-02-12 15:00:38,653 INFO [Listener at localhost/38323] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-12 15:00:38,654 WARN [Listener at localhost/38323] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-12 15:00:38,656 INFO [Listener at localhost/38323] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-12 15:00:38,761 WARN [BP-1881422223-172.31.14.131-1676214000132 heartbeating to localhost/127.0.0.1:35339] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-12 15:00:38,761 WARN [BP-1881422223-172.31.14.131-1676214000132 heartbeating to localhost/127.0.0.1:35339] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1881422223-172.31.14.131-1676214000132 (Datanode Uuid ea05670d-b581-4fa9-ad56-726719504e8a) service to localhost/127.0.0.1:35339 2023-02-12 15:00:38,763 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/dfs/data/data5/current/BP-1881422223-172.31.14.131-1676214000132] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-12 15:00:38,763 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/dfs/data/data6/current/BP-1881422223-172.31.14.131-1676214000132] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-12 15:00:38,765 WARN [Listener at localhost/38323] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-12 15:00:38,767 INFO [Listener at localhost/38323] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-12 15:00:38,871 WARN [BP-1881422223-172.31.14.131-1676214000132 heartbeating to localhost/127.0.0.1:35339] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-12 15:00:38,871 WARN [BP-1881422223-172.31.14.131-1676214000132 heartbeating to localhost/127.0.0.1:35339] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1881422223-172.31.14.131-1676214000132 (Datanode Uuid b02a920e-a537-4a3b-8680-c32dd40f546d) service to localhost/127.0.0.1:35339 2023-02-12 15:00:38,872 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/dfs/data/data3/current/BP-1881422223-172.31.14.131-1676214000132] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-12 15:00:38,873 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/dfs/data/data4/current/BP-1881422223-172.31.14.131-1676214000132] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-12 15:00:38,873 WARN [Listener at localhost/38323] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-12 15:00:38,880 INFO [Listener at localhost/38323] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-12 15:00:38,985 WARN [BP-1881422223-172.31.14.131-1676214000132 heartbeating to localhost/127.0.0.1:35339] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-12 15:00:38,985 WARN [BP-1881422223-172.31.14.131-1676214000132 heartbeating to localhost/127.0.0.1:35339] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1881422223-172.31.14.131-1676214000132 (Datanode Uuid 7dc5048d-e841-4f47-b47b-e95ce3c57560) service to localhost/127.0.0.1:35339 2023-02-12 15:00:38,985 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/dfs/data/data1/current/BP-1881422223-172.31.14.131-1676214000132] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-12 15:00:38,986 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/4542289f-9cc4-2776-8a7e-998cb948d9c3/cluster_cb46f6f8-d855-b581-d43d-e5bb99ac1f00/dfs/data/data2/current/BP-1881422223-172.31.14.131-1676214000132] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-12 15:00:39,013 INFO [Listener at localhost/38323] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-12 15:00:39,141 INFO [Listener at localhost/38323] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-12 15:00:39,221 INFO [Listener at localhost/38323] hbase.HBaseTestingUtility(1293): Minicluster is down